Electronic device and organ contour acquisition method
By constructing three-dimensional image data of the target organ and rendering multiple projection angles on the rendering window, the problem of inaccurate organ calibration in the existing technology is solved, and higher calibration accuracy and matching success rate are achieved.
Patent Information
- Application Number
- CN202110726205.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-29
- Publication Date
- 2026-01-20
- Estimated Expiration
- 2041-06-29
AI Technical Summary
In existing technologies, the two-dimensional model of the target organ does not match the organ captured by the camera accurately, which reduces the accuracy of the target organ calibration. This is especially true in liver surgery, where the liver surface is smooth and has no obvious markers, making it impossible for the camera to obtain depth information and thus impossible to obtain a 3D image.
By constructing 3D image data of the target organ, rendering the 3D image of the target organ on the rendering window according to multiple projection angles, determining the contour corresponding to the projection angle, and performing binarization and downsampling to obtain the contour matching the target organ captured by the camera.
It improves the accuracy of target organ calibration, ensures the precision of contour matching, reduces algorithm complexity, and increases the matching success rate.
Smart Images
Figure CN115546240B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical display technology, and more particularly to an electronic device and a method for acquiring organ contours. Background Technology
[0002] In surgical navigation, it is necessary to register the model of the target organ with the target organ captured by a camera that penetrates deep into the human body, and then display the model. This allows the preoperatively reconstructed blood vessels and lesions to be used to mark the actual locations of blood vessels and lesions in the video.
[0003] For example, in liver surgery, laparoscopy, which involves a camera inserted deep into the body, requires zooming during filming. Therefore, it is impossible to obtain the current camera intrinsic parameters in real time, and thus, it is impossible to obtain the depth information of each pixel. Furthermore, the surface of the liver is very smooth and lacks obvious markers, making it impossible to obtain the depth information of the camera. As a result, the liver image captured by the camera is not a 3D (3-dimensional) image. Instead, it can only be matched with a 2D image captured by the camera and a 2D model of the target organ.
[0004] In existing methods, target organs are acquired through CT images, and a two-dimensional model of the target organ is constructed using the CT images. However, due to the limited shooting angle of CT images, the two-dimensional model of the target organ does not match the organ captured by the camera precisely when it is matched, which reduces the accuracy of subsequent target organ calibration. Summary of the Invention
[0005] This invention provides an electronic device and an organ contour acquisition method. By obtaining a two-dimensional image from any angle through a three-dimensional image, and then extracting the contour, a more matching contour can be found for the calibration of the target organ, thereby improving the accuracy of the calibration.
[0006] In a first aspect, an electronic device provided by an embodiment of the present invention includes: a receiving unit and a processor;
[0007] The receiving unit is used to receive selection instructions triggered by the user;
[0008] The processor is configured to respond to a selection command triggered by a user and determine the target three-dimensional image data of the target organ selected by the selection command; wherein the three-dimensional image formed by the target three-dimensional image data matches the part of the target organ that can be directly captured by a camera;
[0009] For each projection angle, the target 3D image data is rendered on a preset rendering window according to the projection angle to obtain the projected image on the rendering window;
[0010] Based on the projected image on the rendering window, the contour of the target organ corresponding to the projection angle is determined, so as to achieve calibration based on the contour corresponding to each projection angle and the target organ captured by the camera.
[0011] The aforementioned electronic device, after the user selects a target organ, determines the target 3D image data that matches the portion of the target organ that can be directly captured by a camera. Then, according to multiple projection angles, it renders the target 3D image data on a rendering window to obtain a projected image, thereby obtaining the contour of the corresponding target organ. This allows for calibration based on the contour corresponding to each projection angle and the target organ captured by the camera. This invention can obtain a 2D image from any angle from the 3D image of the target organ, so that when its contour is matched with the target organ captured by the camera, a more accurate contour can be found, thus achieving calibration of the target organ and improving the accuracy of calibration.
[0012] In one possible implementation, the processor is specifically used for:
[0013] From the three-dimensional image data of the target organ, the three-dimensional image data corresponding to the parts of the target organ that cannot be directly captured by the camera is removed, and the remaining three-dimensional image data of the target organ is used as the target three-dimensional image data of the target organ. The three-dimensional image data of the target organ is obtained based on multiple two-dimensional medical images containing the target organ.
[0014] Because organs in the human body are stacked together, cameras may not be able to capture the entire organ when taking pictures. To address this, the aforementioned electronic device removes the 3D image data corresponding to parts of the target organ that cannot be directly captured by the camera from the 3D image data constructed from multiple 2D medical images. The remaining 3D image data is then used to create a 3D image that matches the parts that can be directly captured by the camera, thus improving the accuracy of the matching.
[0015] In one possible implementation, the processor is specifically used for:
[0016] A critical value is determined based on the image data along the target axis in the three-dimensional image data of the target organ; wherein, the target axis direction is determined based on the parts of the target organ that cannot be directly captured by a camera; the critical value is a value representing the critical portion between the parts of the target organ that cannot be directly captured by a camera and the parts of the target organ that can be directly captured by a camera.
[0017] The three-dimensional image data of the target organ whose image data in the target axis direction is greater than the threshold value is taken as the target three-dimensional image data of the target organ.
[0018] The aforementioned electronic device provides a method for acquiring three-dimensional image data of a target. It can find a value representing the critical part between the part of the target organ that cannot be directly photographed by a camera and the part of the target organ that can be directly photographed by a camera through the image data in the target axis direction, and thus find the part of the target organ that can be directly photographed by a camera based on this value.
[0019] In one possible implementation, the processor is specifically used for:
[0020] Multiple projection angles are determined based on the shooting angle range of the target organ and the preset movement step size in each axis direction.
[0021] When a camera is used to photograph a target organ, it needs to enter the human body to take the picture, so it cannot capture the target organ from all angles. Based on this, the aforementioned electronic device can determine multiple projection angles according to the shooting angle range of the target organ and the preset step length in each axis direction. This avoids setting projection angles that the camera cannot capture, thus avoiding increasing the workload.
[0022] In one possible implementation, the processor is specifically used for:
[0023] The projected image on the rendering window is binarized;
[0024] Based on the projected image on the rendering window after binarization, the initial contour of the target organ corresponding to the projection angle is determined;
[0025] The initial contour of the target organ corresponding to the projection angle is downsampled to obtain the contour of the target organ corresponding to the projection angle.
[0026] The aforementioned electronic device can binarize the projected image, determine the initial contour of the target organ corresponding to the projection angle based on the binarized projected image, and perform downsampling on the initial contour of the target organ, which can reduce the algorithm complexity during subsequent matching.
[0027] Secondly, an organ contour acquisition method provided by an embodiment of the present invention includes:
[0028] In response to a user-triggered selection command, the system determines the target three-dimensional image data of the target organ selected by the selection command; wherein the three-dimensional image formed by the target three-dimensional image data matches the portion of the target organ that can be directly captured by a camera.
[0029] For each projection angle, the target 3D image data is rendered on a preset rendering window according to the projection angle to obtain the projected image on the rendering window;
[0030] Based on the projected image on the rendering window, the contour of the target organ corresponding to the projection angle is determined, so as to achieve calibration based on the contour corresponding to each projection angle and the target organ captured by the camera.
[0031] In one possible implementation, determining the target three-dimensional image data of the target organ selected by the selection instruction includes:
[0032] From the three-dimensional image data of the target organ, the three-dimensional image data corresponding to the parts of the target organ that cannot be directly captured by the camera is removed, and the remaining three-dimensional image data of the target organ is used as the target three-dimensional image data of the target organ. The three-dimensional image data of the target organ is obtained based on multiple two-dimensional medical images containing the target organ.
[0033] In one possible implementation, the three-dimensional image data of the target organ is used to remove the three-dimensional image data corresponding to parts of the target organ that cannot be directly captured by a camera, and the remaining three-dimensional image data of the target organ is used as the target three-dimensional image data of the target organ, including:
[0034] A critical value is determined based on the image data along the target axis in the three-dimensional image data of the target organ; wherein, the target axis direction is determined based on the parts of the target organ that cannot be directly captured by a camera; the critical value is a value representing the critical portion between the parts of the target organ that cannot be directly captured by a camera and the parts of the target organ that can be directly captured by a camera.
[0035] The three-dimensional image data of the target organ whose image data in the target axis direction is greater than the threshold value is taken as the target three-dimensional image data of the target organ.
[0036] In one possible implementation, multiple projection angles are determined in the following way:
[0037] Multiple projection angles are determined based on the shooting angle range of the target organ and the preset movement step size in each axis direction.
[0038] In one possible implementation, determining the contour of the target organ corresponding to the projection angle based on the projected image on the rendering window includes:
[0039] The projected image on the rendering window is binarized;
[0040] Based on the projected image on the rendering window after binarization, the initial contour of the target organ corresponding to the projection angle is determined;
[0041] The initial contour of the target organ corresponding to the projection angle is downsampled to obtain the contour of the target organ corresponding to the projection angle.
[0042] Thirdly, this application also provides a computer storage medium storing a computer program thereon, which, when executed by a processing unit, implements the steps of the organ contour acquisition method described in the second aspect.
[0043] Furthermore, the technical effects of any of the implementation methods in the second to third aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here.
[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention, but do not constitute an undue limitation of the invention.
[0046] Figure 1 This is a schematic diagram of a preoperative calibration scenario provided by an embodiment of the present invention;
[0047] Figure 2 This is a structural diagram of an electronic device provided in an embodiment of the present invention;
[0048] Figure 3 This is a flowchart of an organ contour acquisition method provided in an embodiment of the present invention;
[0049] Figure 4A This is a schematic diagram of a three-dimensional image of the liver provided in an embodiment of the present invention;
[0050] Figure 4B This is a schematic diagram of another three-dimensional image of the liver provided in an embodiment of the present invention;
[0051] Figure 5 This is a schematic diagram of a contour extraction process corresponding to projection angle 1 provided in an embodiment of the present invention;
[0052] Figure 6 This is a schematic diagram of a contour extraction process corresponding to projection angle 2 provided in an embodiment of the present invention;
[0053] Figure 7 This is a schematic diagram of the display interface of the mitk open-source software platform provided in an embodiment of the present invention. Detailed Implementation
[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0055] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0056] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.
[0057] The camera proposed in this invention is a photographic device capable of capturing images of organs inside the human body.
[0058] In surgical navigation, it is necessary to register the model of the target organ with the target organ captured by a camera that penetrates deep into the human body, and then display the model to mark the actual location of blood vessels and lesions in the video.
[0059] Combination Figure 1 As shown, the doctor selects the target organ to be operated on and constructs a three-dimensional image of the target organ using multiple CT (Computed Tomography) images taken before the operation, thus obtaining three-dimensional image data.
[0060] Because organs in the human body are stacked together, cameras may not be able to capture the entire organ when taking pictures. To address this, we obtain target 3D image data that matches the portion of the target organ that can be directly captured by the camera from the target organ's 3D image data. Then, for different projection angles, we render the target 3D image data on a preset rendering window to obtain the projected image on the rendering window. Based on the projected image on the rendering window, we determine the outline of the target organ corresponding to the projection angle.
[0061] The contours corresponding to multiple projection angles are matched with the target organ captured by the camera to obtain the best matching contour of the target organ captured by the camera. Triangles represent points that need to be marked. The position of the triangle is calculated through the contour, so that surgery can be performed on the triangle position of the target organ.
[0062] For example, the electronic device provided in the embodiments of the present invention will be introduced first. Figure 2 A schematic diagram of the electronic device is shown.
[0063] The following uses an electronic device as an example to illustrate the embodiments in detail. It should be understood that... Figure 2 The electronic device shown is merely an example, and electronic devices can have more than... Figure 2 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.
[0064] Figure 2 The diagram illustrates a hardware configuration block diagram of an electronic device according to an exemplary embodiment. Figure 2 As shown, the electronic device includes components such as a radio frequency (RF) circuit 210, a power supply 220, a processor 230, a memory 240, an input unit 250, a display unit 260, a communication interface 270, and a wireless Fidelity (Wi-Fi) module 280. Those skilled in the art will understand that... Figure 2 The structure of the electronic device shown does not constitute a limitation on the terminal. The electronic device provided in the embodiments of this application may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0065] The RF circuit 210 can be used for data reception and transmission during communication. Specifically, after receiving downlink data from the base station, the RF circuit 210 sends it to the processor 230 for processing; additionally, it sends uplink data to be transmitted to the base station. Typically, the RF circuit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.
[0066] Furthermore, the RF circuit 210 can also communicate wirelessly with networks and other terminals. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).
[0067] Wi-Fi technology is a short-range wireless transmission technology. Electronic devices can connect to an access point (AP) via a Wi-Fi module 280, thereby enabling access to data networks. The Wi-Fi module 280 can be used for receiving and sending data during communication.
[0068] The electronic device can physically connect to other devices via the communication interface 270. Optionally, the communication interface 270 can be connected to the communication interfaces of other devices via a cable to enable data transmission between the electronic device and other devices. Other devices, such as cameras, can input images of organs captured by the camera into the electronic device for display.
[0069] The electronic device can receive images of organs transmitted by the camera via communication interface 270, Wi-Fi module 280, or RF circuit 210, enabling the electronic device to match and display them, such as... Figure 1 The image on the right side of the middle page.
[0070] Since the electronic device in this embodiment is capable of communication services and sending information to other contacts, it needs to have data transmission capabilities, meaning it needs to include a communication module. Although Figure 2 The RF circuit 210, the Wi-Fi module 280, and the communication interface 270 are shown, but it is understood that the electronic device contains at least one of the above components or other communication modules (such as a Bluetooth module) for data transmission.
[0071] The memory 240 can be used to store software programs and modules. The processor 230 executes various functional applications and data processing of the reconstruction device by running the software programs and modules stored in the memory 240. Furthermore, when the processor 230 executes the program code in the memory 240, it can implement the embodiments of the present invention. Figure 3 Part or all of the process.
[0072] Optionally, the memory 240 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, various applications (such as communication applications), and a facial recognition module; the data storage area may store data created based on the use of the terminal (such as various images, video files, and other multimedia files, as well as facial information templates).
[0073] In addition, the memory 240 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0074] The input unit 250 can be used to receive numerical or character information input by the user, and to generate key signal inputs related to user settings and function control.
[0075] Optionally, the input unit 250 may include a touch panel 251 and other input terminals 252.
[0076] The touch panel 251, also known as a touchscreen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 251), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 251 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 230, and can also receive and execute commands from the processor 230. Furthermore, the touch panel 251 can be implemented using various types of touch technologies, such as resistive, capacitive, infrared, and surface acoustic wave.
[0077] Optionally, the other input terminal 252 may include, but is not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0078] The display unit 260 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device. The display unit 260 is the display system of the electronic device, used to present the interface and realize human-computer interaction.
[0079] The display unit 260 may include a display panel 261. Optionally, the display panel 261 may be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
[0080] Furthermore, the touch panel 251 may cover the display panel 261. When the touch panel 251 detects a touch operation on or near it, it transmits the information to the processor 230 to determine the type of touch event. Subsequently, the processor 230 provides corresponding visual output on the display panel 261 according to the type of touch event.
[0081] Although Figure 2 In this embodiment, the touch panel 251 and the display panel 261 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 251 and the display panel 261 can be integrated to realize the input and output functions of the electronic device.
[0082] The processor 230 is the control center of the electronic device. It connects various components through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 240, and calling data stored in the memory 240, it performs various functions of the electronic device and processes data, thereby realizing various services based on the electronic device.
[0083] Optionally, the processor 230 may include one or more processing units. Optionally, the processor 230 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 230.
[0084] The electronic device also includes a power supply 220 (such as a battery) for supplying power to various components. Optionally, the power supply 220 can be logically connected to the processor 230 through a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption.
[0085] The technical solution of the present invention will be described below with reference to the accompanying drawings.
[0086] Combination Figure 3 As shown, this embodiment of the invention provides an organ contour acquisition method, applied to the electronic device described above, including:
[0087] S300: Responds to a user-triggered selection command and determines the target 3D image data of the target organ selected by the selection command; wherein, the 3D image formed by the target 3D image data matches the part of the target organ that can be directly captured by the camera.
[0088] S301: For each projection angle, the target 3D image data is rendered on a preset rendering window according to the projection angle to obtain the projected image on the rendering window.
[0089] S301: Based on the projected image on the rendering window, determine the contour of the target organ corresponding to the projection angle, so as to achieve calibration based on the contour corresponding to each projection angle and the target organ captured by the camera.
[0090] In the above embodiments, since the camera needs to enter the human body to capture images of the target organ, and the organs inside the body are stacked on top of each other, the camera may not be able to capture the target organ from all angles. Therefore, this embodiment of the invention obtains the target 3D image data of the target organ that matches the part of the target organ that can be directly captured by the camera, avoiding the need to perform contour matching on the parts that were not captured, which would lead to matching failure. This invention can improve the success rate of subsequent contour matching. At the same time, this invention uses a method of rendering 3D images in a rendering window. Since the angle of the 3D image projection can be adjusted on the rendering window, any face of the 3D image can be displayed on the rendering window, obtaining the projected image of the target organ on any face, i.e., a 2D image. Then, the corresponding contour is obtained based on the 2D image. In this way, when matching the contours corresponding to various projection angles with the target organ captured by the camera, a more accurate matching contour can be found, realizing the calibration of the target organ and thus improving the calibration accuracy.
[0091] The target three-dimensional image data of the target organ selected by the selection instruction includes:
[0092] From the three-dimensional image data of the target organ, the three-dimensional image data corresponding to the parts of the target organ that cannot be directly captured by the camera is removed, and the remaining three-dimensional image data of the target organ is used as the target three-dimensional image data of the target organ. The three-dimensional image data of the target organ is obtained based on multiple two-dimensional medical images containing the target organ.
[0093] Specifically, the process begins by capturing two-dimensional medical images of the target organ, such as CT scans. A three-dimensional image of the target organ is then created based on this CT image, yielding three-dimensional image data. Next, the three-dimensional image data of the target organ is processed by removing the three-dimensional image data corresponding to parts of the target organ that cannot be directly captured by a camera. This process removes the three-dimensional image data corresponding to parts of the target organ that are obscured by other organs and therefore cannot be directly captured by a camera. This ensures that the remaining three-dimensional image data matches the parts of the target organ that can be directly captured by a camera, thereby improving the accuracy of subsequent contour matching.
[0094] Taking liver imaging as an example, the posterior half of the liver is covered by its inner wall, making it impossible for the camera to capture this area. A 3D image of the liver is created based on CT scans. This 3D image data is generated in a coordinate system determined by the scanning direction of the CT image. The scanning direction of the CT image corresponds to the three axes of the 3D image data: the X-axis (from left to right), the Z-axis (from bottom to top), and the Y-axis (from inside to outside). The posterior half of the liver, covered by its inner wall, is removed along the Y-axis. The remaining 3D image data corresponding to the anterior half of the liver is used as the target 3D image data. Figure 4A and Figure 4B As shown, the white part is the liver portion that is not obscured by the inside, which is the part that can be directly captured by the camera. The part of the liver surrounded by the curve is the part that is obscured by the inner wall, which is the part that cannot be directly captured by the camera.
[0095] Furthermore, based on the image data along the target axis in the three-dimensional image data of the target organ, a critical value is determined; whereby the target axis direction is determined based on the parts of the target organ that cannot be directly captured by a camera; the critical value is a value representing the critical portion between the parts of the target organ that cannot be directly captured by a camera and the parts of the target organ that can be directly captured by a camera.
[0096] The three-dimensional image data of the target organ whose image data in the target axis direction is greater than the threshold value is used as the target three-dimensional image data of the target organ.
[0097] Among them, the three-dimensional image data containing the critical value is the three-dimensional image data corresponding to the critical part, and the critical value is calculated based on the image data in the target axis direction.
[0098] Taking the liver as an example, the back of the liver in the Y-axis direction is covered by the inner wall, which is a part of the liver that cannot be directly photographed by a camera. In other words, the area with a smaller y-axis is the area covered by the inner wall, and the area with a larger y-axis is the area not covered by the inner wall.
[0099] Using the Y-axis as the target axis, all 3D image data are represented as (x, y, z), with the Y-axis data representing the y-values. A critical value is determined based on all y-values; this critical value can be the average of all y-values. The portion of the 3D image data containing this critical value is the critical part, representing the area of the liver that cannot be directly captured by a camera, down to the area that can be directly captured. The average of all y-values represents this critical part. Then, 3D image data of the liver with y-values less than the average (representing areas covered by the inner wall and areas not directly captured by a camera) is removed. 3D image data of the liver with y-values greater than the average (representing areas not covered by the inner wall and areas directly captured by a camera) is used as the target 3D image data for the liver.
[0100] Since the camera captures images of target organs inside the human body, not all angles of the target organ can be captured. For example, when photographing the liver, the area below the liver cannot be captured. Therefore, when obtaining the contours of different projection angles, it is unnecessary to extract the contours of the areas that cannot be captured. To address this, this embodiment of the invention proposes determining multiple projection angles in the following way:
[0101] Multiple projection angles are determined based on the target organ's imaging angle range and the preset movement step size in each axis direction.
[0102] By setting different projection angles, similar to a user rotating a 3D image, different aspects of the 3D image are displayed, resulting in a projected image of the 3D image. In other words, the target 3D image data is rendered on the rendering window according to different projection angles, allowing different aspects of the 3D image to be displayed on the rendering window.
[0103] Taking the liver as an example, in laparoscopic surgery videos, the rotation of the laparoscope along the X-axis will not exceed 60°, along the Y-axis will not exceed 120°, and along the Z-axis will not exceed 60°. In other words, the maximum rotation angles along the X, Y, and Z axes are 60°, 120°, and 60°, respectively. Since the laparoscopic video angle is always upward-facing of the liver, the rotation ranges for the X-axis are set to [-60°, 0°], the Y-axis to [-60°, 60°], and the Z-axis to [-60°, 60°]. Preset movement steps are used in the X, Y, and Z axes; for example, 5°, 5°, and 5° respectively. Projection angles are set, for example, -5° for the X-axis, 0° for the Y-axis, and 0° for the Z-axis. Then, based on the set projection angles, the target 3D image data is rendered onto the rendering window to obtain the corresponding projected image, and the corresponding contour is obtained from the projected image.
[0104] After completing the contour corresponding to the projection angle, keep the X-axis and Y-axis directions unchanged, and increase the Z-axis by 5° to determine the corresponding contour. Repeat this process, using a pre-set movement step size on the Z-axis, to determine all contours within the range of [-60°, 60°]. After determining these contours, keep the X-axis direction unchanged, increase the Y-axis by 5°, and set the Z-axis to 0° to obtain the corresponding contour. Repeat this process, using a pre-set movement step size on the Y-axis, to determine all contours within the range of [-60°, 60°]. Following this pattern, determine the contours corresponding to all projection angles with rotation ranges of [-60°, 0°] on the X-axis, [-60°, 60°] on the Y-axis, and [-60°, 60°] on the Z-axis.
[0105] Furthermore, since the 3D image data is in color, the RGB image of the 3D image is converted into a grayscale image to prepare for the subsequent binarization of the projected image on the rendering window.
[0106] The methods for obtaining the contour include: binarizing the projected image on the rendering window;
[0107] Based on the projected image on the rendering window after binarization, determine the initial contour of the target organ corresponding to the projection angle;
[0108] The initial contour of the target organ corresponding to the projection angle is downsampled to obtain the contour of the target organ corresponding to the projection angle.
[0109] Since the projected image is a grayscale image, it is binarized by setting a threshold, for example, 1, to obtain a binarized projected image. Then, OpenCV's `findContour` function is used to obtain the initial contour of the target organ in the projected image. To reduce the complexity of subsequent algorithms, the Douglas Peuker algorithm is used to downsample the initial contour of the target organ. The Douglas Peuker algorithm uses the distance difference from a point to an edge. This algorithm starts with a coarse simplification, connecting the first and last vertices of the original polyline to a single edge. Then, the distance from all intermediate vertices to this edge is calculated. The vertex farthest from this edge and whose calculated distance is greater than a specified threshold is marked as a keypoint and added to the simplified vertex set. The above process is recursively performed on all simplified vertex sets until the distance from all vertices of the original polyline to the simplified polyline is less than a given threshold, resulting in the downsampled contour. After downsampling, the number of points in the contour is reduced, thereby reducing the computational load during contour matching.
[0110] like Figure 5 As shown, with a projection angle of -5° in the X-axis direction, 0° in the Y-axis direction, and 0° in the Z-axis direction, the target three-dimensional image data of the liver is rendered onto the rendering window based on this projection angle. Then, the projected image is binarized to obtain a binary image, i.e., a black and white image. Finally, the initial contour of the liver in the projected image is obtained through the findContour function of OpenCV.
[0111] Then, the initial contour is downsampled to obtain a contour with fewer points, which is the contour corresponding to projection angle 1.
[0112] like Figure 6 As shown, according to the projection angle 2 with X-axis direction of -10°, Y-axis direction of 10° and Z-axis direction of 0°, the target three-dimensional image data of the liver is rendered onto the rendering window based on the projection angle 2. Then, the projection image is binarized to obtain a binary image, i.e., a black and white image. Then, the initial contour of the liver in the projection image is obtained through the findContour function of OpenCV.
[0113] Then, the initial contour is downsampled to obtain a contour with fewer points, which is the contour corresponding to the projection angle 2.
[0114] Through the above process, the contours corresponding to the projection angles within all shooting ranges can be obtained. For example, if the number of projection angles is n, the contours corresponding to projection angle n can be obtained.
[0115] This allows for contour matching between the contours corresponding to projection angle 1, projection angle 2, ..., projection angle n, and the liver image captured by the camera, obtaining the best-matching contour, which is then used for calibration.
[0116] Based on the above description, this invention provides an application. It can construct three-dimensional images of multiple organs within the human body based on pre-stored CT images of the human body. Combined with... Figure 7 As shown, the 3D image data is imported into the Mitk open-source software platform. The Mitk platform loads the 3D image data and displays the organ names from the 3D image data in a list of candidate organs. The display interface of the Mitk platform allows users to adjust the color of the 3D images of organs and displays data management. In this data management, the name of each organ corresponds to its storage address. The right side of the Mitk platform's display interface displays CT images from different angles.
[0117] In the Mitk open-source software platform, the default background color of the window displaying the 3D image of an organ is gray, and the window itself is surrounded by a yellow line. This window may also display other text or other labels. Furthermore, multiple 3D images of different organs are displayed in the same window, making it impossible to directly distinguish the target organ from all the organ projections on the display interface, thus making it impossible to obtain the projected outline of the target organ.
[0118] If only the projection of the target organ is captured in the window displaying the 3D image of the organ, the background color of the rendering window needs to be manually changed, and the color of the target organ also needs to be modified to distinguish it from the colors of other organs. After the projection is completed, the target organ region still needs to be segmented from the entire projected image. This process is quite tedious and time-consuming.
[0119] To address this, the present invention utilizes VTK to acquire projected images. Specifically, a hidden rendering window is created and not displayed to the user. This prevents any noticeable lag during contour extraction. Furthermore, after extracting one target organ, the user is immediately notified of the extraction process, allowing them to extract contours for other organs and improving the user experience. The rendering window's size is set to match the window displaying the 3D organ image in Mitk. Since the Mitk window displays the same size as the image in the video, the matching and calibration processes do not require resizing. The rendering window is also set to black to prepare for subsequent binarization of the projected image.
[0120] Specifically, when a user selects the target organ for contour extraction, such as the liver, on the display interface of Mitk's open-source software platform, and then clicks the "Get Contour" button, the 3D image data of the liver is obtained through the storage address of the liver recorded on the Mitk open-source software platform. Then, the 3D image data corresponding to the parts of the liver that cannot be directly captured by the camera is removed from the 3D image data of the liver to obtain the target 3D image data of the liver. Then, different projection angles are set, and the target 3D image data of the liver is rendered onto the RenderWindow, i.e., the rendering window, according to the projection image on the RenderWindow. The contour corresponding to the projection angle is determined based on the projection image on the RenderWindow.
[0121] Based on the organ contour acquisition method described above, this embodiment of the invention also provides an electronic device, including: a receiving unit and a processor;
[0122] The receiving unit can be one of the above. Figure 2 The input unit 250 is described in the text.
[0123] The receiving unit is used to receive selection instructions triggered by the user;
[0124] The processor is configured to respond to a selection command triggered by a user and determine the target three-dimensional image data of the target organ selected by the selection command; wherein the three-dimensional image formed by the target three-dimensional image data matches the part of the target organ that can be directly captured by a camera;
[0125] For each projection angle, the target 3D image data is rendered on a preset rendering window according to the projection angle to obtain the projected image on the rendering window;
[0126] Based on the projected image on the rendering window, the contour of the target organ corresponding to the projection angle is determined, so as to achieve calibration based on the contour corresponding to each projection angle and the target organ captured by the camera.
[0127] Optionally, the processor is specifically used for:
[0128] From the three-dimensional image data of the target organ, the three-dimensional image data corresponding to the parts of the target organ that cannot be directly captured by the camera is removed, and the remaining three-dimensional image data of the target organ is used as the target three-dimensional image data of the target organ. The three-dimensional image data of the target organ is obtained based on multiple two-dimensional medical images containing the target organ.
[0129] Optionally, the processor is specifically used for:
[0130] A critical value is determined based on the image data along the target axis in the three-dimensional image data of the target organ; wherein, the target axis direction is determined based on the parts of the target organ that cannot be directly captured by a camera; the critical value is a value representing the critical portion between the parts of the target organ that cannot be directly captured by a camera and the parts of the target organ that can be directly captured by a camera.
[0131] The three-dimensional image data of the target organ whose image data in the target axis direction is greater than the threshold value is taken as the target three-dimensional image data of the target organ.
[0132] Optionally, the processor is specifically used for:
[0133] Multiple projection angles are determined based on the shooting angle range of the target organ and the preset movement step size in each axis direction.
[0134] Optionally, the processor is specifically used for:
[0135] The projected image on the rendering window is binarized;
[0136] Based on the projected image on the rendering window after binarization, the initial contour of the target organ corresponding to the projection angle is determined;
[0137] The initial contour of the target organ corresponding to the projection angle is downsampled to obtain the contour of the target organ corresponding to the projection angle.
[0138] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the organ contour acquisition method described above. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.
[0139] This invention also provides a computer program product that, when run on an electronic device, causes the electronic device to execute any of the organ contour acquisition methods described above in this invention.
[0140] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.
[0141] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. An electronic device, comprising: The method comprises the steps of: receiving a selection instruction triggered by a user; determining target three-dimensional image data of a target organ selected by the selection instruction in response to the selection instruction triggered by the user; wherein a three-dimensional image formed by the target three-dimensional image data matches a part of the target organ that can be directly photographed by a camera; rendering the target three-dimensional image data on a preset rendering window according to each projection angle to obtain a projection image on the rendering window; determining an outline of the target organ corresponding to the projection angle according to the projection image on the rendering window, so as to calibrate the target organ photographed by the camera according to the outline corresponding to each projection angle. The processor is specifically configured to:
2. The electronic device of claim 1, wherein, remove three-dimensional image data corresponding to a part of the target organ that cannot be directly photographed by the camera from three-dimensional image data of the target organ, and take the remaining three-dimensional image data of the target organ as the target three-dimensional image data of the target organ, wherein the three-dimensional image data of the target organ is obtained according to a plurality of two-dimensional medical images containing the target organ. The processor is specifically configured to:
3. The electronic device of claim 2, wherein, determine a critical value according to image data in a target axis direction of the three-dimensional image data of the target organ, wherein the target axis direction is determined according to the part of the target organ that cannot be directly photographed by the camera; the critical value is a value representing a critical part of the part of the target organ that cannot be directly photographed by the camera to the part of the target organ that can be directly photographed by the camera; take the three-dimensional image data of the target organ in the target axis direction greater than the critical value as the target three-dimensional image data of the target organ. The processor is specifically configured to:
4. The electronic device of claim 1, wherein, determine a plurality of projection angles according to a photographing angle range of the target organ and a preset moving step in each axis direction. The processor is specifically configured to:
5. The electronic device according to any one of claims 1 to 4, characterized by perform a binarization processing on the projection image on the rendering window; determine an initial outline of the target organ corresponding to the projection angle according to the projection image on the rendering window after the binarization processing; perform a down-sampling processing on the initial outline of the target organ corresponding to the projection angle to obtain the outline of the target organ corresponding to the projection angle. The method comprises the steps of:
6. A method of organ contouring, the method comprising: determining target three-dimensional image data of a target organ selected by a selection instruction in response to the selection instruction triggered by a user; wherein a three-dimensional image formed by the target three-dimensional image data matches a part of the target organ that can be directly photographed by a camera; rendering the target three-dimensional image data on a preset rendering window according to each projection angle to obtain a projection image on the rendering window; determining an outline of the target organ corresponding to the projection angle according to the projection image on the rendering window, so as to calibrate the target organ photographed by the camera according to the outline corresponding to each projection angle. The method comprises the steps of:
7. The organ contouring method of claim 6, wherein, determining target three-dimensional image data of a target organ selected by a selection instruction in response to the selection instruction triggered by a user; wherein a three-dimensional image formed by the target three-dimensional image data matches a part of the target organ that can be directly photographed by a camera; The three-dimensional image data of the target organ is obtained from a plurality of two-dimensional medical images containing the target organ.
8. The organ contouring method of claim 7, wherein, The three-dimensional image data of the target organ is obtained from a plurality of two-dimensional medical images containing the target organ. The critical value is determined according to the image data in the target axis direction of the three-dimensional image data of the target organ, wherein the target axis direction is determined according to the part of the target organ that cannot be directly photographed by the camera; and the critical value is a value representing the critical part of the part of the target organ that cannot be directly photographed by the camera to the part of the target organ that can be directly photographed by the camera. The three-dimensional image data of the target organ in the target axis direction whose image data is greater than the critical value is taken as the target three-dimensional image data of the target organ.
9. The method of claim 6, wherein, The plurality of projection angles are determined in the following manner: The plurality of projection angles are determined according to the photographing angle range of the target organ and the preset moving step in each axis direction.
10. The method according to any one of claims 6 to 9, characterized in that, The contour of the target organ corresponding to the projection angle is determined according to the projection image on the rendering window, including: The projection image on the rendering window is binarized; The initial contour of the target organ corresponding to the projection angle is determined according to the binarized projection image on the rendering window; The initial contour of the target organ corresponding to the projection angle is down-sampled to obtain the contour of the target organ corresponding to the projection angle.