Processing method, laparoscopic system, and non-volatile storage medium

Abstract

To generate an image suitable for observation.SOLUTION: A laparoscope system includes: a video laparoscope; a camera controller controlling the video laparoscope; a computer which has a frame grabber capturing a laparoscopic image from the video laparoscope; and a screen connected to the computer. The computer analyzes the laparoscopic image with respect to visibility of an object, calculates an arrangement of a 3D model of the object generated in advance with respect to a position of the object in the laparoscopic image, generates a visual expression of the 3D model, compares the visual expression of the 3D model with at least one of color information and brightness information on the laparoscopic image at the 3D model position, adjusts the visual expression of the 3D model on the basis of the result of comparison in an area where the difference is below prescribed threshold, and generates a composite image by using the laparoscopic image and the adjusted visual expression of the 3D model.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a processing method and a laparoscope system for processing laparoscopic images captured during a laparoscopic procedure using a video laparoscope, and a non - volatile storage medium containing instructions for a computer.

Background Art

[0002] In a laparoscopic procedure using a video laparoscope, an operator views the image generated by the video laparoscope on a screen which may be a primary OP monitor. In some cases, the laparoscopic image presented to the operator is enhanced by overlaying the laparoscopic image with a rendering of a 3D model of the organ or other organic structure being operated on. Such a 3D model is pre - generated based on a prior radiological examination of the patient and serves to assist in orientation and movement during the procedure.

[0003] Such an overlay of the 3D model covers portions of the live surgical image depending on its position and orientation. If the similarity between the structures shown in the two parts of the composite image is excessive, the overlay may be difficult to distinguish from the actual laparoscopic image.

[0004] In the case of a transparent or semi - transparent overlay, the overlaid 3D model of the organ visually blends with the laparoscopic image. This can lead to a loss of detail or, in extreme cases, confusion between the background live surgical image and the overlaid display of specific details.

[0005] Another effect of the overlay is that the overlaid image of the 3D model of the organ may block the view of surgical instruments within the laparoscopic image.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] The object of the present invention is to provide a processing method and a laparoscopy system that can generate images suitable for observation. [Means for solving the problem]

[0008] Such objectives can be addressed by a processing method for processing laparoscopic images taken during a laparoscopic procedure, which includes the steps of: analyzing the laparoscopic image with respect to the visibility of the object using a video laparoscope; calculating the placement of a pre-generated 3D model of the object relative to the position of the object in the laparoscopic image; generating a visual representation of the 3D model; comparing at least one of the color information and brightness information of the laparoscopic image at the position of the 3D model with the visual representation of the 3D model; adjusting the visual representation of the 3D model based on the comparison results in areas where the compared difference falls below a predetermined threshold; and generating a composite image using the laparoscopic image and the adjusted visual representation of the 3D model.

[0009] The composite image may be displayed on a screen.

[0010] In this embodiment, laparoscopic images are processed in real time.

[0011] The object on which the 3D model is used may be at least one of one or more organs and organic structures.

[0012] In the processing method described herein, the 3D model of the object is not only rendered as an overlay on the laparoscopic image, but is also adjusted to have greater contrast with respect to the underlying image, thus achieving the goal of better assisting the operator by reducing the risk of confusion between the visualization of the 3D model and the actual laparoscopic image. This is done by identifying areas with insufficient color or brightness contrast and adjusting the visualization of the 3D model in those areas, thereby increasing the contrast, particularly with respect to color and / or brightness.

[0013] In the embodiment, at least one of the comparison and adjustment of the visual representation of the 3D model is performed at the pixel level and at least one of the multi-pixel segments of the laparoscopic image. Pixel-level comparison and / or adjustment yields the most detailed composite image, while the use of multi-pixel segments may be used to increase the repetition rate of the enhanced laparoscopic image, among other things, as it is computationally less time-consuming.

[0014] Segmentation in comparison may differ from segmentation in adjusting the visual representation of a 3D model. For example, segments in adjusting the visual representation of a 3D model may be circular or elliptical and completely encompass the segments used for comparison. This latter characteristic helps to make the rendering of the 3D model more natural and avoids arbitrary raster effects.

[0015] In this embodiment, the brightness of the laparoscopic image may be reduced, and / or the brightness of the visual representation of the 3D model may be increased. If the brightness of the laparoscopic image is reduced and the brightness of the visual representation of the 3D model is increased, the contrast between the visualization of the 3D model and the laparoscopic image is further increased.

[0016] Further embodiments include detecting surgical instruments in the laparoscopic image and adjusting the visual representation of the 3D model so as not to obscure the surgical instruments. This may be done by making the visualization of the 3D model more transparent in the region of the laparoscopic image containing the surgical instruments, or by cutting out the portion of the organ obscured by the surgical instruments from the rendering of the 3D model of the object. This makes the surgical instruments appear to be in front of the 3D model of the object in perspective. Detection of surgical instruments may be done, for example, by at least one of object recognition and edge detection.

[0017] In further embodiments, for example, in a further display mode, details from the inner region of the 3D model of the object may be excluded, only the outer contour of the 3D model may be rendered, or predetermined edges of the 3D model including the outer contour of the 3D model may be rendered.

[0018] The outer contour of a 3D model may be outlined with two highly contrasting colors or luminances. This ensures that the contour appears equal against light, dark, and mid-tone backgrounds. In color images, the contrasting colors may be chosen to represent complementary colors, specifically those that have the greatest contrast with the dominant color in the laparoscopic image.

[0019] A further aspect of the present invention is a laparoscopic system for processing laparoscopic images captured during a laparoscopic procedure using a video laparoscope. The laparoscopic system includes a video laparoscope, a camera controller configured to control the video laparoscope, a computer having a frame grabber configured to capture laparoscopic images from the video laparoscope, and a screen connected to the computer. The computer analyzes the laparoscopic image with respect to the visibility of an object, calculates the placement of a pre-generated 3D model of the object relative to the position of the object in the laparoscopic image, generates a visual representation of the 3D model, and compares at least one of the visual representation of the 3D model and the color information and luminance information of the laparoscopic image at the position of the 3D model. The representation of the 3D model is adjusted based on the result of the comparison in a region where the compared difference is below a predetermined threshold, and a composite image is generated using the laparoscopic image and the adjusted visual representation of the 3D model.

[0020] Thereby, the laparoscopic system embodies the same features, characteristics and solutions as the aforementioned processing method.

[0021] In an embodiment, the computer is configured to perform at least one of the comparison and adjustment of the visual representation of the 3D model at least one of the pixel level of the laparoscopic image and the segment of a plurality of pixels.

[0022] In one embodiment, the computer is configured to perform at least one of reducing the luminance of the laparoscopic image and increasing the luminance of the visual representation of the 3D model.

[0023] In one embodiment, the computer is configured to detect a surgical instrument in the laparoscopic image and adjust the visual representation of the 3D model so as not to obscure the surgical instrument.

[0024] In one embodiment, the computer is configured to detect the surgical instrument by at least one of recognition of the object and edge detection.

[0025] The computer may be configured to exclude details from the inner region of the 3D model of the object.

[0026] In one embodiment, the computer is configured to render at least one of the outer contour of the 3D model and a predetermined edge of the 3D model that includes the outer contour of the 3D model.

[0027] In another embodiment, the computer may outline the outer contour of the 3D model with two highly contrasting colors or luminances.

[0028] The above-described embodiments and functions may be in the configuration of the computer's frame grabber for respective purposes, particularly when the frame grabber is provided with an image processing function.

[0029] Another aspect of the present invention resides in a non-volatile storage medium including instructions configured to cause a computer to execute the above-described processing method. Such instructions may be loaded into the computer of the laparoscope system or its frame grabber described above.

[0030] Further features will become apparent from the description of the embodiments together with the claims and the accompanying drawings. The embodiments can satisfy individual features or combinations of some features.

[0031] Based on exemplary embodiments, without limiting the general intention of the present invention, in the embodiments described below, reference is explicitly made to the drawings with respect to all detailed disclosures not described in more detail in the text.

Brief Description of the Drawings

[0032] [Figure 1] It is a diagram showing a synthetic image of a laparoscopic procedure generated according to a known method. [Figure 2] It is a diagram showing a synthetic image of a laparoscopic procedure generated according to the first embodiment. [Figure 3]This is a schematic diagram of one embodiment of a laparoscopic system for processing laparoscopic images. [Figure 4] This figure shows a composite image of a laparoscopic procedure generated according to the second embodiment. [Figure 5] This figure shows a composite image of a laparoscopic procedure generated according to the third embodiment. [Figure 6] This is a schematic diagram of a machine learning model for use in the present invention. [Modes for carrying out the invention]

[0033] In drawings, elements of the same or similar type, or their corresponding parts, are given the same reference number to avoid the need to reintroduce items.

[0034] Figure 1 shows a composite image 6 of a laparoscopic procedure, generated according to a known method, which can be displayed on a central surgical monitor in the operating room. A 3D model 4 of the organic structure is rendered and overlaid on the live laparoscopic image 2, thereby highlighting important organic structures within the surgical area. The 3D model 4 is generated, for example, based on pre-operative radiography of the patient.

[0035] Two areas, highlighted by square frames in the composite image 6 of Figure 1, are due to shortcomings in the method used to render the 3D model 4. The area labeled "1" contains portions of the background laparoscopic image 2 that have color and / or brightness values ​​very similar to the rendering of the 3D model 4 of the organic structure, making it difficult to distinguish between the 3D model 4 and the actual laparoscopic image 2. The other area labeled "2" shows that the rendering of the 3D model 4 obstructs the view of the surgical instruments 8 currently being used to manipulate the organic structure within the surgical area. Any of these types of effects can be irritating to the surgeon.

[0036] Figure 2 shows a composite image 6 of a laparoscopic procedure generated according to the first embodiment. In this first standard embodiment, a composite image 6 is shown of a laparoscopic image 2 and an overlay 3D model 4 of organs and other organic structures. In the 3D model 4, different organic structures are represented in different colors to highlight their distinction from one another.

[0037] The rendering of 3D model 4 is based on radiographic data of organs and organic structures obtained from radiographic examinations prior to laparoscopic procedures. The rendering of 3D model 4 is generated according to the method of the present invention, and the rendering of 3D model 4 is compared pixel by pixel with the laparoscopic image 2, and the visual representation of color and / or brightness is adjusted to be more clearly distinguishable in areas where the difference in color and / or brightness was originally too small. This enhancement allows the surgeon to clearly understand which parts of the composite image 6 represent the live laparoscopic image 2 and which parts represent 3D model 4.

[0038] Furthermore, the comparison between 3D model 4 and laparoscopic image 2 may be performed not only at the pixel level, but also at the segment level, which consists of multiple pixels. The same applies to the adjustment of the visual representation of 3D model 4. Here, the segments used in the adjustment may differ from the segments used in the comparison. Moreover, the segments used in the adjustment may be circular or elliptical and may completely encompass the segments used in the comparison.

[0039] Figure 3 shows a schematic diagram of one embodiment of a laparoscopic system 10 for processing laparoscopic images 2. The laparoscopic system 10 includes a source of laparoscopic images 2, i.e., a video laparoscope 11, and is connected to a camera controller 12 designed and configured to control the operation of the video laparoscope 11 and to receive image data of laparoscopic images 2 from the video laparoscope 11. The camera controller 12 is also connected to a computer 14 having a frame grabber 16, which may be implemented in software connected to the computer 14 or as an inserted hardware card. The frame grabber 16 is designed and configured to process individual frames of laparoscopic images 2. Processing of individual frames of laparoscopic images 2 may, in some cases, be performed within the frame grabber 16 or within the computer 14.

[0040] The computer 14, or optionally the frame grabber 16, has in its memory a 3D model 4 of the patient's organs or organic structures being examined using the video laparoscope 11, and is configured to match the position and orientation of the 3D model 4 with the live surgical laparoscopic image 2. Once the position and orientation of the 3D model 4 relative to the laparoscopic image 2 is established, software running on the computer 14 or the frame grabber 16 may combine the laparoscopic image 2 with a rendering of the 3D model 4 to form a composite image 6, which is then displayed on the screen 18.

[0041] Figure 4 shows a composite image 6 of a laparoscopic procedure generated according to a second embodiment of the method according to the present invention. The basic steps performed in the method described with respect to Figure 2 remain the same, except that the surgical instrument 8 is detected in the laparoscopic image 2. To give the surgeon a complete and unobstructed view, the vicinity of the surgical instrument 8 is defined by a notched region 20 and removed from the rendering of the 3D model 4, thus clarifying the view of the surgical instrument 8 and its surroundings. The remainder of the rendering of the 3D model 4 provides the surgeon with visual information regarding the orientation and position of the surgical area displayed in the laparoscopic image 2.

[0042] Figure 5 shows a composite image 6 of a laparoscopic procedure generated according to the third embodiment. In this case, the 3D model 4 of the organic structure is simply overlaid on the real-time laparoscopic image 2 as a contour, in this case as a high-contrast contour 22. The high-contrast contour 22 has the form of double lines of different colors, and these two lines are directly adjacent to each other. In this case, one line is white and the other is black, as white and black provide the sharpest possible contrast. Such a high-contrast contour 22 is visible on any background structure, with the white line being prominent in low-luminance areas and the black line being prominent in high-luminance areas. In addition to, or instead of, the black and white lines, colored lines with high color contrast may be used. The colors may also be selected to have the best possible color contrast against the red background of a typical laparoscopic image 2.

[0043] The high-contrast contour 22 of the third embodiment shown in Figure 5 may be an alternative representation of the 3D model 4 to those shown in Figures 2 and 4.

[0044] Figures 2, 4, and 5 also show further differences and modifications that can be used to improve the contrast between the representation of 3D model 4 and the background laparoscopic image 2, such as increasing the brightness of the rendering of 3D model 4, decreasing the brightness of the background laparoscopic image 2, or increasing the contrast and / or color contrast in one of the two images, such as the rendering of 3D model 4.

[0045] Figure 6 shows an exemplary schematic diagram of a machine learning model 30 configured to recognize objects such as organs, organic structures, or surgical instruments in a laparoscopic image 2 provided as input features by a video laparoscope 11. In various embodiments, the machine learning model 30 includes an input interface 32 to which patient-specific 3D model data is provided as input to an artificial intelligence (hereinafter referred to as AI) model 34, and a processor that performs inference operations to which the laparoscopic image 2 and the 3D model data are applied to the AI ​​model 34 to generate organs or organic structures represented in the 3D model 4, and optionally the position and orientation of surgical instruments 8 that can be seen in the laparoscopic image 2. Based on the output of the AI ​​model 34, a composite image 6 of the laparoscopic image 2 overlaid with a rendering of the 3D model 4 can be generated and shown on a screen 18 to a user, for example, a clinician, and the 3D model 4 may be refined when it detects surgical instruments 8 in the laparoscopic image 2.

[0046] In some embodiments, the input interface 32 may be a direct data link between the machine learning model 30 and one or more medical devices that generate at least some of the input features. For example, the input interface 32 may directly transmit laparoscopic images 2 to the machine learning model 30 during a therapeutic and / or diagnostic medical procedure. Additionally or alternatively, the input interface 32 may be a classic user interface that facilitates interaction between the user and the machine learning model 30. For example, the input interface 32 may consist of a user interface from which the user can manually input laparoscopic images 2. Additionally or alternatively, the input interface 32 may provide the machine learning model 30 with access to an electronic patient record from which one or more laparoscopic images or 3D model data may be extracted as input features. In any of these cases, the input interface 32 is configured to collect one or more input features associated with a particular patient at or before the time the machine learning model 30 is used to assess the presence, location, and orientation of organs, organic structures, and / or surgical instruments.

[0047] Based on one or more of the input features described above, the processor uses the AI ​​model 34 to perform inference operations to generate the system output described above. For example, the input interface 32 may deliver laparoscopic features to the input layer of the AI ​​model 34, which propagates these input features to the output layer via the AI ​​model 34. The AI ​​model 34 can provide the laparoscopic system 10 with the ability to perform tasks without being explicitly programmed by inferring based on patterns discovered in the analysis of the data. The AI ​​model 34 explores the study and construction of algorithms (e.g., machine learning algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by constructing the AI ​​model 34 from exemplary training data to make data-driven predictions or decisions, which are expressed as outputs or evaluations.

[0048] Machine learning (hereinafter referred to as ML) has two common modes: supervised ML and unsupervised ML. Supervised ML learns the relationship between inputs and outputs using prior knowledge (for example, correlating inputs with outputs or results). The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between training inputs and outputs, so that the ML model can implement the same relationship when given inputs to produce the corresponding outputs. Unsupervised ML is the training of ML algorithms using unclassified and unlabeled information, allowing the algorithm to act on that information without guidance. Unsupervised ML is useful for exploratory analysis because it can automatically identify structures within the data.

[0049] Common tasks in supervised machine learning are classification and regression problems. Classification problems, also known as categorization problems, aim to classify items into one of several categorical values ​​(e.g., is this object an apple or an orange?). Regression algorithms aim to quantify several items (for example, by assigning a score to some input values). Some examples of commonly used supervised machine learning algorithms include logistic regression (LR), naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM).

[0050] Some common tasks for unsupervised ML include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised ML algorithms are K-means clustering, principal component analysis, and autoencoders.

[0051] Another type of machine learning is federative learning (also known as collaborative learning), which trains algorithms across multiple decentralized devices that hold local data without exchanging data. This approach contrasts with traditional centralized machine learning techniques where all local datasets are uploaded to a single server, as well as classical decentralized methods that often assume local data samples are uniformly distributed. Federative learning enables multiple parties to build a common, robust machine learning model without sharing data, and thus allows for addressing critical issues such as data privacy, data security, data access rights, and access to heterogeneous data.

[0052] In some examples, the AI ​​model 34 may be trained continuously or periodically prior to the execution of inference operations by the processor running the AI ​​model 34. Then, during the inference operation, patient-specific input features provided to the AI ​​model 34 may propagate from the input layer, through one or more hidden layers, to output layers corresponding to the location, orientation, and type of structures ultimately found in the laparoscopic image. For example, the AI ​​model 34 may identify the type, location, and orientation of a particular organ or organic structure represented by the patient-specific 3D model 4, or the location and, if applicable, the type and / or orientation of a surgical instrument 8 in the laparoscopic image 2.

[0053] During and / or following the inference operation, the location, type, and / or orientation of the object thus found may be communicated to the user via a user interface (UI), particularly by rendering a correspondingly positioned and oriented 3D model 4. The AI ​​model 34 may also have output information that helps define the notch region 20 in rendering the 3D model 4 of the composite image 6 in the presence of the surgical instrument 8.

[0054] While embodiments of the present invention have been shown and described, it will be understood that various modifications and changes to the form or detail can be readily made without departing from the spirit of the invention. Accordingly, the present invention is intended to be configured to encompass all modifications that may be included within the scope of the appended claims, rather than being limited to the exact forms described and illustrated. [Explanation of symbols]

[0055] 2. Laparoscopic images 4 3D models 6. Composite image 8 Surgical instruments 10 Laparoscopic Systems 11. Video Laparoscopy 12 Camera Controllers 14 Computers 16 Frame Grabber 18 screens 20 Notch area 22 High-Contrast Contours 30 Machine Learning Models 32 Input Interfaces 34 AI Models 36 Output Interfaces

Claims

1. A method for processing laparoscopic images taken during a laparoscopic procedure using a video laparoscope, The steps include: acquiring the laparoscopic image of the object, The steps include: calculating the placement of the 3D model relative to the laparoscopic image such that the corresponding feature positions of the pre-generated 3D model of the object match the feature positions of the object in the laparoscopic image; The steps include generating a visual representation of the 3D model, The steps include comparing at least one of the color information and brightness information at corresponding positions between the visual representation of the 3D model and the laparoscopic image, A step of adjusting the visual representation of the 3D model based on the results of the comparison in the region where the difference is below a predetermined threshold, A processing method comprising the steps of generating a composite image using the laparoscopic images and the adjusted visual representation of the 3D model.

2. The processing method according to claim 1, further comprising the step of displaying the composite image on a screen.

3. The processing method according to claim 1, further comprising the step of processing the laparoscopic images in real time.

4. The processing method according to claim 1, wherein the object is at least one of one or more organs and organic structures.

5. The processing method according to claim 1, wherein at least one of the comparison step and the adjustment step of the 3D model is performed on at least one of the pixels of the laparoscopic image and a segment composed of multiple pixels.

6. The processing method according to claim 5, wherein the segment in the adjustment step and the segment in the comparison step of the visual representation of the 3D model are different.

7. The processing method according to claim 6, wherein the segment in the adjusting step of the visual representation of the 3D model is circular or elliptical, and the segment in the adjusting step completely encompasses the segment in the comparing step.

8. The processing method according to claim 1, comprising at least one of reducing the brightness of the laparoscopic image and increasing the brightness of the visual representation of the 3D model.

9. The steps include detecting surgical instruments in the laparoscopic image, The processing method according to claim 1, further comprising the step of adjusting the visual representation of the 3D model so as not to obscure the surgical instruments.

10. The processing method according to claim 9, wherein the step of detecting the surgical instrument in the laparoscopic image is to detect the surgical instrument by at least one of object recognition and edge detection.

11. The processing method according to claim 1, comprising at least one of the following steps: excluding details from the inner region of the 3D model of the object; rendering only the outer contour of the 3D model; and rendering predetermined edges of the 3D model including the outer contour of the 3D model.

12. The processing method according to claim 11, wherein in the step of rendering only the outer contour of at least one of the 3D models or the step of rendering a predetermined edge of the 3D model, the outer contour of the 3D model is outlined with two contrasting colors or luminances.

13. A laparoscopic system for processing laparoscopic images taken during a laparoscopic procedure using a video laparoscope, The aforementioned video laparoscope, A camera controller configured to control the video laparoscope, A computer having a frame grabber configured to capture laparoscopic images from the video laparoscope, The computer has a screen connected to it, Here, the computer is The laparoscopic image of the object is obtained, The placement of the 3D model relative to the laparoscopic image is calculated such that the corresponding feature positions of the pre-generated 3D model of the object match the feature positions of the object in the laparoscopic image. A visual representation of the aforementioned 3D model is generated, The steps include comparing at least one of the color information and brightness information at corresponding positions between the visual representation of the 3D model and the laparoscopic image, Based on the results of the comparison in the region where the difference is below a predetermined threshold, the visual representation of the 3D model is adjusted. A composite image is generated using the laparoscopic images and the adjusted visual representation of the 3D model. A laparoscopic system configured as follows.

14. The laparoscopic system according to claim 13, wherein the computer is configured to perform at least one of the steps of comparing and adjusting the visual representation of the 3D model for at least one of the pixels of the laparoscopic image and a segment composed of multiple pixels.

15. The laparoscopic system according to claim 13, wherein the computer is configured to reduce the brightness of the laparoscopic image and to increase the brightness of the visual representation of the 3D model.

16. The laparoscopic system according to claim 13, wherein the computer is configured to detect surgical instruments in the laparoscopic image and to adjust the visual representation of the 3D model so as not to obscure the surgical instruments.

17. The laparoscopic system according to claim 16, wherein the computer is configured to perform the detection of the surgical instrument by at least one of object recognition and edge detection.

18. The laparoscopic system according to claim 13, wherein the computer is configured to exclude details from the internal region of the 3D model of the object.

19. The laparoscopic system according to claim 13, wherein the computer is configured to render the outer contour of the 3D model and at least one of predetermined edges of the 3D model including the outer contour of the 3D model.

20. The laparoscopic system according to claim 13, wherein the computer is configured to outline the outer contour of the 3D model with two or more contrasting colors or luminances.

21. A non-volatile storage medium comprising a computer instruction configured to cause the computer to execute the processing method described in claim 1.

Images