Endoscope image display system and endoscope image display method

Abstract

The present specification discloses an endoscope image display system and an endoscope image display method. The system comprises an endoscope, an endoscope tracer, a space register, a positioning device, a control device and a display device, the endoscope tracer is fixed to the endoscope, and the space register is fixed to a patient; the positioning device is used for acquiring a first spatial relationship between the spatial registrar and the endoscope tracer and sending the first spatial relationship to the control equipment; the endoscope is used for acquiring an image of a diseased part of a patient through an endoscope lens to obtain an endoscope image and sending the endoscope image to the control equipment; the control equipment is used for determining the rotation angle of the endoscope lens relative to the endoscope channel, determining the pose information of the endoscope image relative to the patient according to the rotation angle and the first spatial relationship, and sending the pose information to the display equipment; the display device is used for displaying the endoscope image based on the pose information.

Description

Technical Field

[0001] This manual relates to the field of major surgical procedures, and in particular to an endoscopic image display system and an endoscopic image display method. Background Technology

[0002] Currently, whether it is orthopedics, neurology, or interventional surgery, minimally invasive surgery is required for treatment. The requirements for minimally invasive and precise surgical operations are becoming increasingly higher. Due to the high precision and stability of robot-guided surgery, it is beginning to be widely used in actual surgical operations.

[0003] Minimally invasive surgeries based on robot navigation usually require the use of an endoscope. An endoscope typically consists of two parts: an optical channel and a camera. Operators can adjust the direction of the displayed image by rotating the interface between the optical channel and the camera.

[0004] However, because the operator of the surgical robot adjusts the endoscope's lens, the endoscopic image they see may deviate from its actual orientation within the patient's body. This makes it impossible for the operator to determine which direction the endoscope is actually pointing towards the patient, which side of the image represents the patient's head and feet, and which side represents the left and right. Furthermore, it makes it impossible to intuitively adjust the endoscope's position. This severely impacts the precision of the surgical procedure.

[0005] Therefore, how to help operators accurately position the endoscope inside the patient's body to ensure the precision of the surgical procedure is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides an endoscopic image display system and an endoscopic image display method to partially solve the aforementioned problems existing in the prior art.

[0007] The following technical solution is adopted in this specification:

[0008] This specification provides an endoscopic image display system, the system comprising: an endoscope, an endoscope tracer, a spatial register, a positioning device, a control device, and a display device, wherein the endoscope tracer is fixed to the endoscope, and the spatial register is fixed to the patient;

[0009] The positioning device is used to obtain a first spatial relationship between the spatial register and the endoscope tracer, and send the first spatial relationship to the control device.

[0010] The endoscope is used to acquire images of the affected area of ​​the patient through the endoscope lens, obtain endoscopic images, and send the endoscopic images to the control device;

[0011] The control device is used to determine the rotation angle of the endoscope lens relative to the endoscope channel, and based on the rotation angle and the first spatial relationship, determine the pose information of the endoscope image relative to the patient, and send the pose information to the display device.

[0012] The display device is used to display the endoscopic image based on the pose information.

[0013] Optionally, the endoscopic image display system further includes: an image acquisition unit;

[0014] The spatial registrar includes: a patient tracker and an image indicator;

[0015] The image acquisition device is specifically used to acquire a three-dimensional image of the image indicator and a three-dimensional image of the affected area of ​​the patient and send them to the control device;

[0016] The positioning device is specifically used to determine the spatial relationship between the endoscope tracer and the patient tracer, and send the first spatial relationship to the control device.

[0017] The control device is specifically used to: determine a second spatial relationship between the image indicator and the patient based on the three-dimensional image of the image indicator and the three-dimensional image of the diseased area; determine a third spatial relationship between the patient and the endoscope based on the first spatial relationship and the second spatial relationship; and determine the pose information of the endoscope image relative to the patient based on the rotation angle and the third spatial relationship.

[0018] Optionally, the endoscopic image is provided with a direction marker;

[0019] The control device is specifically used to determine the rotation angle of the endoscope lens relative to the endoscope channel based on the direction corresponding to the direction mark and the preset reference direction.

[0020] Optionally, the control device is specifically used to determine a fourth spatial relationship between the endoscopic image and the endoscope based on the rotation angle;

[0021] Based on the first spatial relationship and the fourth spatial relationship, a fifth spatial relationship between the endoscopic image and the patient is determined;

[0022] Based on the fifth spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

[0023] Optionally, the display device is specifically used to display the pose information on the endoscopic image.

[0024] Optionally, the display device is specifically used to correct the initial pose information of the endoscope image using the pose information to obtain corrected pose information;

[0025] The corrected pose information is displayed on the endoscopic image.

[0026] Optionally, the display device is specifically used to adjust the pose of the endoscopic image based on the pose information, and to display the adjusted endoscopic image on a three-dimensional image of the patient's affected area.

[0027] This manual provides a method for displaying endoscopic images, including:

[0028] Acquire endoscopic images of the affected area of ​​the patient obtained by the endoscope, and determine a first spatial relationship between the spatial register and the endoscopic tracer, wherein the endoscopic tracer is fixed on the endoscope and the spatial register is fixed on the patient;

[0029] Determine the rotation angle of the endoscope lens relative to the endoscope channel, and determine the pose information of the endoscope image relative to the patient based on the rotation angle and the first spatial relationship;

[0030] Based on the pose information, the endoscopic image is displayed.

[0031] Optionally, the spatial registrar includes: a patient tracer and an image indicator;

[0032] Determining the first spatial relationship between the spatial registrar and the endoscopic tracer specifically includes:

[0033] The spatial relationship between the endoscopic tracer and the patient tracer is determined as the first spatial relationship;

[0034] Acquire a three-dimensional image of the image indicator and a three-dimensional image of the affected area of ​​the patient, and determine a second spatial relationship between the image indicator and the patient based on the three-dimensional image of the image indicator and the three-dimensional image of the affected area;

[0035] Based on the rotation angle and the first spatial relationship, the pose information of the endoscopic image relative to the patient is determined, specifically including:

[0036] Based on the first spatial relationship and the second spatial relationship, a third spatial relationship between the patient and the endoscope is determined;

[0037] Based on the rotation angle and the third spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

[0038] Optionally, determining the rotation angle of the endoscope lens relative to the endoscope channel specifically includes:

[0039] The rotation angle of the endoscope lens relative to the endoscope channel is determined based on the direction corresponding to the direction mark set on the endoscope image and the preset reference direction.

[0040] Optionally, based on the rotation angle and the first spatial relationship, the pose information of the endoscopic image relative to the patient is determined, specifically including:

[0041] Based on the rotation angle, a fourth spatial relationship between the endoscopic image and the endoscope is determined;

[0042] Based on the first spatial relationship and the fourth spatial relationship, a fifth spatial relationship between the endoscopic image and the patient is determined;

[0043] Based on the fifth spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

[0044] Optionally, displaying the endoscopic image based on the pose information specifically includes:

[0045] The pose information is displayed on the endoscopic image.

[0046] Optionally, displaying the endoscopic image based on the pose information specifically includes:

[0047] The initial pose information of the endoscopic image is corrected using the pose information to obtain the corrected pose information.

[0048] The corrected pose information is displayed on the endoscopic image.

[0049] Optionally, displaying the endoscopic image based on the pose information specifically includes:

[0050] Based on the pose information, the pose of the endoscopic image is adjusted, and the adjusted endoscopic image is stacked on the three-dimensional image of the patient's diseased area for display.

[0051] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0052] The endoscopic image display system provided in this specification includes: an endoscope, an endoscope tracer, a spatial register, a positioning device, a control device, and a display device. The endoscope tracer is fixed to the endoscope, and the spatial register is fixed to the patient. The positioning device is used to acquire a first spatial relationship between the spatial register and the endoscope tracer, and send the first spatial relationship to the control device. The endoscope is used to acquire images of the affected area of ​​the patient through the endoscope lens to obtain an endoscopic image, and send the endoscopic image to the control device. The control device is used to determine the rotation angle of the endoscope lens relative to the endoscope channel, and based on the rotation angle and the first spatial relationship, determine the pose information of the endoscopic image relative to the patient, and send the pose information to the display device. The display device is used to display the endoscopic image based on the pose information.

[0053] As can be seen from the above method, when displaying endoscopic images, this solution can determine the pose information of the endoscopic image relative to the patient based on the first spatial relationship between the spatial register and the endoscopic tracer and the rotation angle of the endoscopic lens relative to the endoscopic channel. In this way, even if the direction of the endoscopic image is adjusted by rotating the endoscopic lens during the operation, the operator can still determine its actual direction relative to the patient based on the pose information of the endoscopic image relative to the patient and perform the operation, thus ensuring the accuracy of the surgical operation. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 This is a schematic diagram of an endoscope image before and after adjustment, as provided in this instruction manual.

[0056] Figure 2 This is a schematic diagram of an endoscopic image display system provided in this specification;

[0057] Figure 3 This is a schematic diagram of a modified endoscopic image provided in this instruction manual;

[0058] Figure 4 This is a schematic diagram of a modified endoscopic image provided in this instruction manual;

[0059] Figure 5 This is a schematic diagram of a modified endoscopic image provided in this instruction manual;

[0060] Figure 6 This is a schematic diagram of a modified endoscopic image provided in this instruction manual;

[0061] Figure 7 This is a schematic diagram of a modified endoscopic image provided in this instruction manual;

[0062] Figure 8 This is a flowchart illustrating a method for displaying endoscopic images provided in this specification;

[0063] Figure 9 For one application and provided in this specification Figure 8 A schematic diagram of the control equipment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0065] In robotic-guided surgery, operators typically use the robotic arm to control the endoscope to perform surgical procedures. The endoscope usually consists of two parts: the endoscope channel (optical channel) and the endoscope lens (camera lens). The end of the endoscope channel points to the lesion location, and the object being imaged is transmitted to the lens through the endoscope channel. The lens then captures the image and uploads it to the display device.

[0066] The endoscopic camera lens and channel are detachable, and the interface is rotatable for adjusting the orientation of the displayed image. During surgery, the operator can rotate the interface to obtain the desired image orientation. For ease of understanding, this manual provides a schematic diagram of the endoscopic image before and after adjustment, as shown below. Figure 1 As shown.

[0067] Figure 1 This is a schematic diagram of an endoscope image before and after adjustment, as provided in this instruction manual.

[0068] in, Figure 1 (Left) shows the endoscopic image aligned with the direction of the patient's body. At this time, the endoscope lens and the endoscope channel are aligned, the top of the endoscopic image corresponds to the direction of the patient's head, and the bottom of the endoscopic image corresponds to the direction of the patient's feet.

[0069] Figure 1 (Right) shows the endoscopic image obtained after rotating the endoscope lens. At this time, the endoscopic image also rotates, and the direction of the endoscopic image deviates from the actual direction of the patient's body. However, the spatial relationship between the endoscopic channel and the patient remains unchanged.

[0070] The triangular marker (direction marker) in the diagram indicates the rotation direction of the endoscope lens. When the endoscope lens and the endoscope image are deflected, the direction marker will also deflect accordingly.

[0071] In traditional endoscopic surgery, the doctor stands beside the patient's bed, holding an endoscope. The doctor knows the approximate position and direction of the endoscope, but the direction of the image movement after moving the endoscope is inconsistent with the direction of the endoscope movement: for example, if the doctor moves the endoscope to the right, the endoscope image moves upward.

[0072] In robotic-guided surgery or remote surgery, doctors can only determine the location of images through endoscopic images. Experienced doctors can observe human tissue structures to determine the approximate location, but it is more difficult for doctors with less experience.

[0073] In order to accurately determine the actual position of the images acquired by the endoscope relative to the patient during the surgical procedure and to ensure the precision of the surgery, the technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0074] Figure 2 This is a schematic diagram of an endoscopic image display system provided in this specification. The endoscopic image display system includes: an endoscope, an endoscopic tracer, a spatial register, a positioning device, a control device (not shown in the figure), and a display device (not shown in the figure).

[0075] The endoscope tracer is fixed on the endoscope, the control device can be a server or a control terminal, and the display device can be a monitor with image display function.

[0076] The spatial register is fixed near the patient's affected area. The spatial register is equipped with an image indicator and a patient tracker, and the relative positions between the two are fixed. The image indicator can be equipped with at least three black metal spheres that can perform three-dimensional imaging.

[0077] At least three positioning light source markers can be set on the endoscope tracer and the patient tracer. The relative positions of each positioning light source marker are fixed. The positioning light source marker can be an infrared light emitting device, a light emitting diode (LED), or an optical marker ball such as a passive reflective sphere.

[0078] The positioning device can acquire the position of each tracer, determine the relative position and angular relationship between each tracer, thereby obtaining the first spatial relationship between the spatial register and the endoscope tracer, and send the first spatial relationship to the control device so that the control device can further determine the spatial relationship between the endoscope and the patient.

[0079] In addition, the endoscopic image display system provided in this manual may also be equipped with an image acquisition device, which may be a computed tomography (CT) machine or other image acquisition equipment.

[0080] The relative positions between the image indicator and the patient tracer are fixed, and their spatial relationship is known, which can be expressed as:

[0081] P 图像指示器 =M 图像指示器to病患示踪器 *P 病患示踪器

[0082] In this specification, P can represent pose information (i.e., spatial coordinates), and M can represent the spatial transformation matrix.

[0083] The positioning device can obtain the spatial relationship between itself and the patient tracker, as well as the spatial relationship between the positioning device and the endoscopic tracker. The spatial relationship between the positioning device and the patient tracker can be expressed as:

[0084] P 定位装置 =M 病患示踪器to定位装置 *P 病患示踪器

[0085] The spatial relationship between the positioning device and the endoscopic tracer can be expressed as:

[0086] P 定位装置 =M 内窥镜示踪器to定位装置 *P 内窥镜示踪器

[0087] This positioning device can further determine a first spatial relationship between the endoscopic tracer and the patient tracer, which can be expressed as:

[0088] P 病患示踪器 =(M 病患示踪器to定位装置 ) -1 M 内窥镜示踪器to定位装置 *P 内窥镜示踪器 The positioning device can then send the initial spatial relationship to the control device.

[0089] In addition, after the spatial registration device is fixed to the patient's body, a three-dimensional image of the affected area and a three-dimensional image of the image indicator can be acquired by an image acquisition device, and then the three-dimensional image of the affected area and the three-dimensional image of the image indicator can be sent to the control device.

[0090] The controller can construct a second spatial relationship between the patient and the image indicator based on the three-dimensional image of the affected area and the three-dimensional image of the image indicator. This second spatial relationship can be represented as follows:

[0091] P 病患=M 图像指示器to病患 *P 图像指示器

[0092] The relative positions of the endoscopic tracer and the endoscope are fixed, and their spatial relationship is known, which can be expressed as:

[0093] P 内窥镜示踪器 =M 内窥镜示踪器to内窥镜 *P 内窥镜

[0094] The control device can determine a third spatial relationship between the patient and the endoscope based on the first spatial relationship, the second spatial relationship, and the aforementioned known spatial relationships. This third spatial relationship can be expressed as:

[0095] P 病患 =M 内窥镜to病患 *P 内窥镜

[0096] in,

[0097] M 内窥镜to病患 =M 图像指示器to病患 *M 图像指示器to病患示踪器 *(M 病患示踪器to定位装置 ) -1 *M 内窥镜示踪器to定位装置 *M 内窥镜示踪器to内窥镜

[0098] When the endoscope lens and endoscope channel are aligned, the position on the endoscopic image is at the six o'clock position, which can therefore be used as a reference direction for the endoscope lens. This reference direction can also be the initial direction of the endoscopic image when the endoscope lens is not rotated.

[0099] When the endoscope lens is rotated, the endoscopic image rotates, and the direction of the orientation markers changes, but the spatial relationship between the endoscope channel and the patient remains unchanged.

[0100] Therefore, the control device can determine the rotation angle of the endoscope lens relative to the endoscope channel based on the direction corresponding to the direction mark on the current endoscope image and the preset reference direction. At this time, the rotation angle of the endoscope lens relative to the endoscope channel is equal to the rotation angle of the endoscope image.

[0101] The control device can determine the fourth spatial relationship between the endoscopic image and the endoscope based on the rotation angle of the endoscope lens relative to the endoscopic channel. This fourth spatial relationship can be expressed as:

[0102] P 内窥镜 =M 内窥镜通道末端to内窥镜 *M 旋转角度 *P 内窥镜影像

[0103] Among them, the endoscopic image corresponds to the image of the lesion at the end of the endoscopic channel, and the relative positional relationship between the end of the endoscopic channel and the endoscope is fixed.

[0104] The control device can then determine a fifth spatial relationship between the endoscopic image and the patient based on the third spatial relationship determined by the first spatial relationship, and the fourth spatial relationship. This fifth spatial relationship can be expressed as:

[0105] P 病患 =M 内窥镜图像to病患 *P 内窥镜影像

[0106] in,

[0107] M 内窥镜影像to病患 =M 内窥镜to病患 *M 内窥镜通道末端to内窥镜 *M 旋转角度

[0108] The control terminal can then determine the position and orientation information of the endoscopic image relative to the patient, i.e., the direction of the endoscopic image relative to the patient's body, based on the fifth spatial relationship.

[0109] For example, when the endoscope lens and endoscope channel are aligned (without rotating the endoscope image), the top of the endoscope image is aligned with the direction of the patient's head, and the bottom is aligned with the direction of the patient's feet. However, when the endoscope lens is rotated 180° relative to the endoscope channel, the bottom of the endoscope image is aligned with the direction of the patient's head, and the top is aligned with the direction of the patient's feet.

[0110] The control terminal can send the pose information to the display device so that the display device can display the endoscopic image based on the pose information.

[0111] In this instruction manual, endoscopic images can be displayed in several ways, including:

[0112] Method 1: The display device can correct the initial pose information of the endoscopic image using pose information to obtain corrected pose information, which is then displayed on the endoscopic image. For ease of understanding, this manual provides a schematic diagram of the corrected endoscopic image, as shown below. Figure 3 As shown.

[0113] Figure 3 This is a schematic diagram of an endoscopic image provided in this instruction manual.

[0114] Where H represents the direction of the patient's head, F represents the direction of the patient's feet, L represents the left side of the patient, and R represents the right side of the patient. Figure 3 (Left) is the endoscopic image with the lens not rotated. Figure 3 (Right) is the endoscopic image after the lens is rotated.

[0115] In the right image, the top of the image is located between the patient's head and right side, but is closer to the head direction, so it is corrected to HR, indicating that the patient's head is slightly to the right; the right side of the image is located between the patient's right side and feet, but is closer to the right side, so it is corrected to RF, indicating that the patient's right side is slightly to the feet; the bottom of the image is located between the patient's feet and left side, but is closer to the feet direction, so it is corrected to FL, indicating that the patient's feet are slightly to the left; the left side of the image is located between the patient's left side and head, but is closer to the left side, so it is corrected to LH, indicating that the patient's left side is slightly to the head.

[0116] Method 2: The display device can directly display the correct pose information on the endoscopic image, such as... Figure 4 As shown.

[0117] Figure 4 This is a schematic diagram of an endoscopic image provided in this instruction manual.

[0118] The endoscopic images are accurately displayed on the endoscopic screen from all directions relative to the patient.

[0119] Method 3: The display device can adjust the pose of the endoscopic image based on pose information and stack the adjusted endoscopic image on the three-dimensional image of the patient's affected area for display. For ease of understanding, this specification provides a schematic diagram of the stacked display of endoscopic images and a three-dimensional image, such as... Figure 5 As shown.

[0120] Figure 5 This is a schematic diagram of an endoscopic image provided in this instruction manual.

[0121] Specifically, when the mouse is pressed and moved up and down over the endoscopic image, the endoscopic image is zoomed; when the mouse is pressed and zoomed over a non-endoscopic image, the 3D image is zoomed. Alternatively, on a touchscreen computer, pinching and stretching two fingers over the endoscopic image zooms in and out, and pinching and stretching two fingers over the endoscopic image zooms in and out, resulting in a 3D image zoom.

[0122] Method 4: The display device displays the 3D image of the affected area and the endoscopic image in different areas, such as... Figure 6 As shown.

[0123] Figure 6 This is a schematic diagram of an endoscopic image provided in this instruction manual.

[0124] The display device can show a three-dimensional image of the affected area on the left and an endoscopic image on the right. The position of the endoscopic channel is indicated on the three-dimensional image (the image shows a yellow cat paw graphic), and the direction of the three-dimensional image is consistent with the direction of the endoscopic image.

[0125] Method 5: The display device can display a 3D image and a patient model at the four corners of the endoscopic image. As the endoscope moves or rotates, the 3D image and patient model move accordingly. For ease of understanding, this manual provides a schematic diagram of the 3D image and patient model displayed at the four corners of the endoscopic image, as shown below. Figure 7 As shown.

[0126] Figure 7 This is a schematic diagram of an endoscopic image provided in this instruction manual.

[0127] The 3D images and human models can also be displayed in other corners, and can be configured according to the doctor's needs.

[0128] Of course, Method 1 or Method 2 can be combined with any of the display methods from Method 3 to Method 5, that is, while displaying the endoscopic image on the 3D model of the lesion, the positional information of the endoscopic image relative to the lesion can be marked on the endoscopic image.

[0129] Furthermore, this specification also provides a method for displaying endoscopic images, such as... Figure 8 As shown.

[0130] Figure 8 This is a flowchart illustrating an endoscopic image display method provided in this specification, including the following steps:

[0131] S801: The control device acquires the endoscopic image of the patient's diseased site collected by the endoscope, and determines the first spatial relationship between the spatial register and the endoscopic tracer, wherein the endoscopic tracer is fixed to the endoscope and the spatial register is fixed to the patient.

[0132] S802: Determine the rotation angle of the endoscope lens relative to the endoscope channel, and determine the pose information of the endoscope image relative to the patient based on the rotation angle and the first spatial relationship;

[0133] S803: Display the endoscopic image based on the pose information.

[0134] Specifically, the control device can determine the spatial relationship between the endoscope tracker and the patient tracker as the first spatial relationship, acquire the three-dimensional image of the image indicator and the three-dimensional image of the lesion site of the patient, and determine the second spatial relationship between the image indicator and the patient based on the three-dimensional image of the image indicator and the three-dimensional image of the lesion site; determine the third spatial relationship between the patient and the endoscope based on the first spatial relationship and the second spatial relationship; and then determine the pose information of the endoscope image relative to the patient based on the rotation angle and the third spatial relationship.

[0135] The control device can determine the fourth spatial relationship between the endoscopic image and the endoscope based on the rotation angle; determine the fifth spatial relationship between the endoscopic image and the patient based on the first and fourth spatial relationships; and determine the pose information of the endoscopic image relative to the patient based on the fifth spatial relationship.

[0136] As can be seen from the above method, this scheme can intuitively and accurately represent the relative positional relationship between the endoscope, endoscopic images, and the patient. Based on this scheme, doctors can accurately determine the position of the endoscope inside the patient's body, which can help less experienced doctors locate the endoscopic image and avoid misoperation caused by incorrect positioning; it can also help experienced doctors quickly locate lesions, and with the help of the robotic system, the endoscope can be accurately moved and rotated.

[0137] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 8 A method for displaying endoscopic images is provided.

[0138] This instruction manual also provides Figure 9 The one shown corresponds to Figure 8 A schematic structural diagram of the control equipment. (For example...) Figure 9 At the hardware level, the control device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business logic. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 8 The endoscopic image display method described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0139] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0140] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0141] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0142] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0143] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0149] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0150] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0154] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. An endoscopic image display system and an endoscopic image display method, characterized in that, The system includes: an endoscope, an endoscope tracer, a spatial register, a positioning device, a control device, and a display device. The endoscope tracer is fixed to the endoscope, and the spatial register is fixed to the patient. The positioning device is used to obtain a first spatial relationship between the spatial register and the endoscope tracer, and send the first spatial relationship to the control device. The endoscope is used to acquire images of the affected area of ​​the patient through the endoscope lens, obtain endoscopic images, and send the endoscopic images to the control device; The control device is used to determine the rotation angle of the endoscope lens relative to the endoscope channel, and based on the rotation angle and the first spatial relationship, determine the pose information of the endoscope image relative to the patient, and send the pose information to the display device. The display device is used to display the endoscopic image based on the pose information.

2. The endoscopic image display system as described in claim 1, characterized in that, The endoscopic image display system also includes: an image acquisition unit; The spatial registrar includes: a patient tracker and an image indicator; The image acquisition device is specifically used to acquire a three-dimensional image of the image indicator and a three-dimensional image of the affected area of ​​the patient and send them to the control device; The positioning device is specifically used to determine the spatial relationship between the endoscope tracer and the patient tracer, and send the first spatial relationship to the control device. The control device is specifically used to: determine a second spatial relationship between the image indicator and the patient based on the three-dimensional image of the image indicator and the three-dimensional image of the diseased area; determine a third spatial relationship between the patient and the endoscope based on the first spatial relationship and the second spatial relationship; and determine the pose information of the endoscope image relative to the patient based on the rotation angle and the third spatial relationship.

3. The endoscopic image display system as described in claim 1, characterized in that, The endoscopic image is equipped with a direction indicator; The control device is specifically used to determine the rotation angle of the endoscope lens relative to the endoscope channel based on the direction corresponding to the direction mark and the preset reference direction.

4. The endoscopic image display system as described in claim 1, characterized in that, The control device is specifically used to determine a fourth spatial relationship between the endoscope image and the endoscope based on the rotation angle. Based on the first spatial relationship and the fourth spatial relationship, a fifth spatial relationship between the endoscopic image and the patient is determined; Based on the fifth spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

5. The endoscopic image display system as described in claim 1, characterized in that, The display device is specifically used to display the pose information on the endoscopic image.

6. The endoscopic image display system as described in claim 1, characterized in that, The display device is specifically used to correct the initial pose information of the endoscope image using the pose information to obtain corrected pose information. The corrected pose information is displayed on the endoscopic image.

7. The endoscopic image display system as described in claim 1, characterized in that, The display device is specifically used to adjust the pose of the endoscopic image based on the pose information, and to display the adjusted endoscopic image on a three-dimensional image of the patient's affected area.

8. A method for displaying endoscopic images, characterized in that, include: Acquire endoscopic images of the affected area of ​​the patient obtained by the endoscope, and determine a first spatial relationship between the spatial register and the endoscopic tracer, wherein the endoscopic tracer is fixed on the endoscope and the spatial register is fixed on the patient. Determine the rotation angle of the endoscope lens relative to the endoscope channel, and determine the pose information of the endoscope image relative to the patient based on the rotation angle and the first spatial relationship; Based on the pose information, the endoscopic image is displayed.

9. The method as described in claim 8, characterized in that, The spatial registrar includes: a patient tracker and an image indicator; Determining the first spatial relationship between the spatial registrar and the endoscopic tracer specifically includes: The spatial relationship between the endoscopic tracer and the patient tracer is determined as the first spatial relationship; Acquire a three-dimensional image of the image indicator and a three-dimensional image of the affected area of ​​the patient, and determine a second spatial relationship between the image indicator and the patient based on the three-dimensional image of the image indicator and the three-dimensional image of the affected area; Based on the rotation angle and the first spatial relationship, the pose information of the endoscopic image relative to the patient is determined, specifically including: Based on the first spatial relationship and the second spatial relationship, a third spatial relationship between the patient and the endoscope is determined; Based on the rotation angle and the third spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

10. The method as described in claim 8, characterized in that, Determining the rotation angle of the endoscope lens relative to the endoscope channel specifically includes: The rotation angle of the endoscope lens relative to the endoscope channel is determined based on the direction corresponding to the direction mark set on the endoscope image and the preset reference direction.

11. The method as described in claim 8, characterized in that, Based on the rotation angle and the first spatial relationship, the pose information of the endoscopic image relative to the patient is determined, specifically including: Based on the rotation angle, a fourth spatial relationship between the endoscopic image and the endoscope is determined; Based on the first spatial relationship and the fourth spatial relationship, a fifth spatial relationship between the endoscopic image and the patient is determined; Based on the fifth spatial relationship, the pose information of the endoscopic image relative to the patient is determined.

12. The method as described in claim 8, characterized in that, Based on the pose information, the endoscopic image is displayed, specifically including: The pose information is displayed on the endoscopic image.

13. The method as described in claim 8, characterized in that, Based on the pose information, the endoscopic image is displayed, specifically including: The initial pose information of the endoscopic image is corrected using the pose information to obtain the corrected pose information. The corrected pose information is displayed on the endoscopic image.

14. The method as described in claim 8, characterized in that, Based on the pose information, the endoscopic image is displayed, specifically including: Based on the pose information, the pose of the endoscopic image is adjusted, and the adjusted endoscopic image is stacked on the three-dimensional image of the patient's diseased area for display.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 8 to 14.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 8 to 14.

Images