Surgical image processing method and surgical image processing apparatus

Abstract

To provide a surgical image processing method and a surgical image processing apparatus that facilitate the distinction of drawing colors. [Solution] The surgical image processing method includes the steps of: determining a drawing color which is the display color of the target structure; determining a background color which is the display color of pixels or surrounding areas around the target structure; calculating the color difference between the drawing color and the background color; and determining whether the color difference exceeds a predetermined value.

Description

[Technical Field]

[0001] The present invention relates to a surgical image processing method and a surgical image processing apparatus. [Background technology]

[0002] Surgical image processing devices used to assist in surgery, and surgical support systems including such surgical image processing devices have been proposed. For example, Patent Document 1 discloses a surgical support system that identifies features contained in surgical images and makes suggestions to the operator based on the identified features. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 2023-523560 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, conventional surgical image processing devices have the problem that it can be difficult to distinguish between drawing colors and background colors, or to distinguish between different drawing colors. Therefore, the present invention aims to provide a surgical image processing method and a surgical image processing device that facilitate the distinction of drawing colors. [Means for solving the problem]

[0005] The surgical image processing method of the present invention includes the steps of: determining a drawing color which is the display color of a target structure; determining a background color which is the display color of pixels or surrounding areas of the target structure; calculating the color difference between the drawing color and the background color; and determining whether the color difference exceeds a predetermined value. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a surgical image processing method and a surgical image processing apparatus that facilitate the distinction of drawing colors. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows the drawing color and background color in the L*a*b* color space. [Figure 2] Figure 2 shows the color information of the entire evaluation image integrated into the L*a*b* color space. [Figure 3] Figure 3 shows the background color determined in relation to the base color. [Figure 4A] Figure 4A shows the drawing color and background color in the L*a*b* color space. [Figure 4B] Figure 4B is an enlarged view of the framed area A1 in Figure 4A. [Figure 5] Figure 5 is a flowchart showing the processing flow of the surgical image processing method of Embodiment 1. [Figure 6] Figure 6 is a block diagram showing the configuration of the surgical image processing device of Embodiment 1. [Figure 7] Figure 7 shows an example of an HSV space where the drawing color is divided into cyan, green, and other colors. [Figure 8] Figure 8 shows the HSV space used to explain how drawing colors are changed. [Figure 9] Figure 9 is a block diagram showing the configuration of the surgical image processing apparatus of Embodiment 3. [Modes for carrying out the invention]

[0008] (Embodiment 1) Hereinafter, the surgical image processing method and the surgical image processing apparatus 1 according to Embodiment 1 of the present invention will be described with reference to the drawings. The surgical image processing method of the present embodiment particularly relates to the determination of the drawing color used when drawing the inferred part for use in a surgical image system. Drawing the inferred part is called inference drawing. As an example of a surgical image system, there is a surgical support system that uses machine learning or AI technology to recognize anatomical structures during surgery (such as connective tissue, nerves, pancreas, nerves, blood, surgical instruments, etc.) and provides an easy-to-see display of the structures for the surgeon to assist the surgeon's recognition. In the present embodiment, the drawing color is determined so as not to be difficult to distinguish from the background color. First, an overview of the selection of the drawing color will be described.

[0009] (Drawing color) The drawing color is the color used when performing inference drawing. The drawing color is usually generated by a specified color, a base color, and a transparency. The specified color is the color specified by the user or the system specifications during drawing. The base color is the original color of the pixel to be drawn. The original color means the original color of the pixel to be drawn. The transparency is the degree of transmitting the drawing color set under predetermined conditions.

[0010] (Base color) When determining the drawing color, the base color can be determined, for example, as follows (1) to (3). (1) Cover the surgical field or the anatomical structures included in the surgical field, and obtain the distribution of the base color from an appropriate number of surgical images in which the distribution of the base color is stable. (2) Determine one or more representative regions that characterize the obtained distribution. (3) Generate the base color based on the color of the determined representative region.

[0011] (Background color) This section explains background color. Background color refers to the color of pixels surrounding a pixel that has been drawn using inference, or the color of the region surrounding that pixel. If there is a color difference between the drawing color and the background color, it becomes easier to distinguish between them. The background color can be determined by finding the color of the pixels or region surrounding the pixel that will become the background color from the surgical image used when generating the background color.

[0012] Convert the foreground color and background color to a color space with uniform color difference. An example of a color space with uniform color difference is L * a * b * It is a color space.

[0013] Calculate the Euclidean distance between the coordinates where the drawing color is closest to the background color. If there are multiple drawing colors, calculate the Euclidean distance for each drawing color to the coordinates closest to the background color. The Euclidean distance represents the color difference. The color difference (delta E) can be calculated, for example, using the following formula. Chromatic difference delta E = √(L1 - L2) 2 +|a1-a2|) 2 +|b1-b2| 2 ) In the above equation, L1, a1, and a2 represent the coordinates of the drawing color, while L2, a2, and b2 represent the coordinates of the background color.

[0014] For example, for all drawing colors, a color difference measurement, or color difference calculation, is performed, and a drawing color is selected that has a color difference that allows it to be recognized as a different color from the background color. In order to further improve the accuracy of the selection of drawing colors as described above, the process for selecting drawing colors may be as follows. In the above process, all background colors were treated as a single unit. In other words, one color was treated as the background color. In contrast, the following process treats multiple colors as the background color.

[0015] First, for the base color, a representative color region is set. The representative color region refers to a region that shows a characteristic distribution of the base color. This region may also be referred to as the base color region as the characteristic region of the distribution. The representative color region will be described later. Next, the color of the pixels around the pixels included in each representative color region, or the color of the region around the pixels included in each representative color region is obtained.

[0016] Then, the drawing color and the background color corresponding to each representative region are converted into a color space with color difference uniformity, for example, the L * a * b * color space, and the color difference delta E, which is the Euclidean distance of the coordinates where each drawing color is closest to the background color, is obtained. By this, only the background color related to the base color can be accurately measured.

[0017] When the width of the distribution of the drawing color is narrow, etc., by treating one color as the background color, it is possible to calculate a color difference that is actually almost in agreement. On the other hand, when increasing the transparency of the drawing color, etc., the chroma of the drawing color decreases, and the drawing color and the background color are likely to overlap. Even in such a case, by obtaining the background color according to the base color, it becomes possible to calculate an accurate color difference.

[0018] Hereinafter, regarding this embodiment, it will be described in the order of the prior art and the technology of this embodiment. (Prior Art and Its Problems) When superimposing and displaying the inference result of an AI model such as a connective tissue recognition AI model on the image before inference, the drawing color is generated based on the specified color, which is the color specified according to the user or system specifications, and the base color. When displaying a single color as the drawing color, it is desirable to select a color that is different from the background in terms of at least one of hue, chroma, or lightness. By doing so, it becomes easier to distinguish from the background. Here, the background refers to, for example, the surgical field.

[0019] In contrast, the transparency of a single color used for drawing may be changed depending on the inference confidence level or setting parameters. In this case, the display is optimized to provide a natural look that does not interfere with the surgery.

[0020] When displaying in this manner, the drawing color changes significantly depending on the transparency, and depending on the background color, it may become difficult to distinguish between the drawing color and the background color. Also, when different colors are assigned to multiple structures and displayed simultaneously, depending on the transparency and background color, the different drawing colors may appear similar, making it difficult to distinguish which structure each color represents.

[0021] Furthermore, the color of the surgical field exhibits a certain bias depending on the surgical procedure and the situation during surgery. Therefore, the colors that can be used as drawing colors within the color scheme are limited. As a result, it is not easy to ensure that the drawing color of the inference result does not overlap with the background color of the surgical field. Moreover, it is even more difficult to use colors that do not overlap, especially when displaying multiple drawing colors simultaneously.

[0022] As described above, when displaying drawing colors with varying transparency depending on the inference confidence level or setting parameters, the combination of transparency and background color can result in the drawing color and background color being very similar, making them difficult to distinguish. However, the number of possible combinations of transparency and background color is enormous. Therefore, selecting an appropriate drawing color is not easy. The same applies when assigning different colors to multiple structures and displaying them simultaneously. Thus, it is necessary to establish a quantitative evaluation method for selecting appropriate colors.

[0023] In this embodiment, the drawing color and background color are plotted in a color space with uniform color difference. A drawing color is selected that has a certain distance from the background color that allows it to be recognized as a different color. When multiple colors are selected as the drawing color, all selectable colors are plotted in the same color space. A color is then selected as the drawing color that has a distance from all of these colors that allows it to be recognized as a different color.

[0024] Furthermore, color selection may be carried out not only by color itself, but also by methods such as limiting transmittance. It is also preferable to verify color distance based on a certain number of inferred images. This makes it easier to cover the color tones of different surgical procedures, regions, and endoscopic systems.

[0025] The following explains the selection of drawing colors with reference to the diagram. Figure 1 shows L * a * b * This figure shows the background and drawing colors in a color space. Figure 1 was generated based on 1600 surgical images, each with different endoscopic systems or target structures. In other words, Figure 1 plots the drawing and background colors based on the aforementioned 1600 surgical images.

[0026] In the example shown in Figure 1, eight drawing colors are shown, from drawing color C1 to drawing color C8. The notation "drawing color" is omitted for drawing colors C2 to C8. Also in Figure 1, the distribution of background colors C101 is shown. The distribution of background colors C101 is sometimes simply referred to as background color C101.

[0027] As shown in Figure 1, each drawing color C1 through C8 does not overlap with the background color C101. Within each drawing color's line, the distance between the background color C101 and the drawing color, i.e., the color difference delta E, is measured at the point closest to the background color C101. In the example shown in Figure 1, point P1 in drawing color C8 is the point closest to the background color C101. Therefore, the distance between point P1 and the background color C101 is calculated. Note that the drawing color changes depending on the background color. Therefore, the above calculation of the color difference delta E may be performed for several representative background colors.

[0028] The method described above evaluates all background colors together as a single background color C101. The actual background color is linked to the base color. The background color can be described as a collection of adjacent pixels on a continuous image relative to the base color. Therefore, the color difference delta E is measured between the drawing color obtained for several representative base colors and the background color obtained from pixels surrounding the pixels from which the base color was obtained, or the background color obtained from the region surrounding the pixels from which the base color was obtained. This makes it possible to adopt the drawing color with higher accuracy.

[0029] The representative color range will be explained with reference to Figures 2 to 4B. As explained earlier, in order to improve the accuracy of selecting drawing colors, instead of treating the background color as a single unit, multiple background colors may be set according to the characteristics of the base color.

[0030] As a prerequisite for setting multiple background colors, multiple feature regions are set for each characteristic of the distribution of the background color. Figure 2 shows L * a * b * This figure shows the color information of all evaluation images integrated into the color space. Here, "all evaluation images" refers to the 1600 surgical images mentioned earlier.

[0031] Figure 2 shows an example where three feature regions are set as feature regions in the distribution of background color. The three feature regions are designated as the first feature region R1, the second feature region R2, and the third feature region R3. In the example in Figure 2, the frequency of occurrence is 10 2 Three feature regions were set on the outer edge of the distribution shown above. Note that the number of feature regions is not limited to three. Furthermore, the positions where the feature regions are set are not particularly limited. It is preferable to set the feature regions at positions that indicate the characteristics of the background color distribution.

[0032] As shown in Figure 2, the background color of the first feature region R1 is set to background color 1, the background color of the second feature region R2 is set to background color 2, and the background color of the third feature region R3 is set to background color 3.

[0033] This section explains how to determine the background color. Figure 3 shows the background color determined in relation to background color 1. When multiple background colors are set, multiple background colors are determined corresponding to each background color. This is because the background color is linked to the background color.

[0034] Figure 3 shows the first background color C111, which corresponds to the background color 1 of the first feature region R1. The background colors can be similarly determined for background colors 2 and 3.

[0035] Next, we will explain an example of how to determine the color difference between the drawing color and the background color, referring to Figures 4A and 4B. Figure 4A shows L * a * b * This diagram shows the drawing color and background color in a color space. Figure 4B is an enlarged view of the framed area A1 in Figure 4A.

[0036] The drawing color C11 shown in Figure 4A is the color determined by the AI ​​analysis model to represent the results of the AI ​​analysis. An example of the AI ​​analysis model is the connective tissue recognition AI model. The drawing color C11 is a cyan-based color.

[0037] Furthermore, the first background color C111 shown in Figure 4A is the background color corresponding to the base color 1 explained earlier. The color difference between the drawing color and the background color is evaluated at the closest distance between the drawing color C11 and the first background color C111, as indicated by arrow A11. As shown in Figure 4B, point P11 is the point on the drawing color C11 closest to the first background color C111. Also, point P21 is the point on the first background color C111 closest to the drawing color C11. As shown in Figure 4B, the closest distance between the drawing color C11 and the first background color C111 is distance D1. The background color is the color obtained by evaluating a 5x5 pixel region with the coordinates of the closest distance as the center point.

[0038] Referring to Figure 5, the processing flow of the surgical image processing method of this embodiment will be described. Figure 5 is a flowchart showing the processing flow of the surgical image processing method of Embodiment 1. In Figure 5 and the following description, S1 means step 1. Also, the processing flow of the surgical image processing method shown in Figure 5 is an example. The surgical image processing method can be modified as appropriate. Other steps can be added to the flow shown in Figure 5, some steps can be removed, and the content of the steps can be changed.

[0039] (S1) Step S1 is the step in which the specified color is determined. In S1, the specified color is determined by the user or manufacturer setting it.

[0040] (S2) Step S2 is the step of recognizing anatomical structures. In S2, for example, an AI model recognizes the anatomical structures during surgery. In the surgical image processing method of this embodiment, parts of the recognized anatomical structures are inferred and drawn.

[0041] (S3) Step S3 is the step for determining the background color. In S3, for example, the distribution of background colors is determined from surgical images, representative regions that characterize the distribution are determined, and background colors are generated based on the colors of the representative regions.

[0042] (S4) S4 is the step that determines the drawing color. The drawing color is generated, for example, by a specified color, a background color, and transparency. Inference drawing is performed using the drawing color determined in S4.

[0043] (S5) S5 is the step that determines the background color. In S5, the color of the area around the pixels to be inferred and drawn is determined.

[0044] (S6) Step S6 is the step to calculate the closest color difference between the foreground color and the background color. In S6, the Euclidean distance between the coordinates where the foreground color and the background color are closest is calculated.

[0045] (S7) Step S7 is a step in which it is determined whether the color difference calculated in S6 is greater than or equal to a predetermined value. The predetermined value can be, for example, a value that makes it easy to distinguish between the drawing color and the background color. If the determination in S7 is NO, the step returns to S4. In S4, the drawing color is determined again. At this time, the drawing color is determined to be a different color, for example, by changing the specified color or transparency. At this time, warning information such as "The specified color does not meet the color difference requirement. It is a difficult color to recognize." can be displayed to the user who set the specified color. In addition, proactive encouragement to the user to take action may be given, such as "Please consider changing the specified color."

[0046] If S7's judgment is YES, the processing flow ends. This is because the drawing color is considered an appropriate color that is easily distinguishable from the background color.

[0047] The surgical image processing apparatus 1 of this embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the surgical image processing apparatus 1 of this embodiment. The surgical image processing apparatus 1 includes, for example, an anatomical structure recognition unit 10, a background color determination unit 12, a drawing color determination unit 14, a background color determination unit 16, a color difference calculation unit 20, and a color difference judgment unit 22.

[0048] The anatomical structure recognition unit 10 is the part that performs step S2, which recognizes the anatomical structure described above. The anatomical structure recognition unit 10 may receive anatomical images from an imaging sensor, such as an endoscope.

[0049] The background color determination unit 12 is responsible for executing step S3, which determines the background color. The drawing color determination unit 14 is responsible for executing step S4, which determines the drawing color. The background color determination unit 16 is responsible for executing step S5, which determines the background color. The color difference calculation unit 20 is responsible for executing step S6, which calculates the closest color difference between the drawing color and the background color. The color difference judgment unit 22 is responsible for executing step S7, which determines whether the color difference calculated in S6 is greater than or equal to a predetermined value.

[0050] Each of the above-mentioned parts may be physically distinct. If the parts are physically distinct, data may be transmitted and received between them as needed. Alternatively, each of the above-mentioned parts may be included in one or more arithmetic processing units, such as a CPU (Central Processing Unit).

[0051] Furthermore, in addition to the parts described above, the surgical image processing device 1 may also include, for example, an inference unit 30 that includes an inference model for analyzing surgical images acquired by a camera such as an endoscope, and a drawing unit 32 that reflects the analysis results by the inference model onto the surgical image and draws the image with the drawing color determined as described above.

[0052] As described above, the surgical image processing method and surgical image processing device of Embodiment 1 enable surgical support that recognizes anatomical structures during surgery and displays them in a way that is easy for the surgeon to see. For example, in a surgical image recognition support program, a connective tissue recognition AI model can provide a display that is easier for the surgeon to see.

[0053] Conventional technologies attempt to improve visibility by coloring target structures, but there are doubts as to whether the color is truly easy to see. Furthermore, overlaying a color different from the target structure's original color can actually make the structure harder to see. Additionally, the target structure's color may be difficult to distinguish from its surroundings. In particular, coloring a target structure can sometimes cause it to blend in with its surroundings, making it harder to see.

[0054] According to the technology of this embodiment, for example, the AI ​​analysis display color of a target structure can be made to be a highly visible color tone that is clearly distinguishable to humans from the surrounding background color. This is because the technology of this embodiment analyzes the AI ​​analysis display color and the surrounding background color, calculates the difference between the two colors, sets the display color, and then displays it.

[0055] (Embodiment 2) The surgical image processing method and surgical image processing apparatus 1 of Embodiment 2 will be described with reference to Figure 7. The following description will focus on the differences from Embodiment 1. Matters not specifically described can be the same as in Embodiment 1. Embodiment 1 mainly concerned the selection of image colors. In contrast, Embodiment 2 mainly concerns the assignment of image colors.

[0056] The outline of the surgical image processing method of Embodiment 2 will now be described. First, multiple target structures are classified into two or more groups of structures that are infrequently used simultaneously or are unlikely to be used simultaneously. In other words, structures that are likely to appear simultaneously are assigned to different color groups. Specifically, for example, connective tissue and nerves are classified into different structural groups because they are frequently shown simultaneously. In contrast, the ureter and pancreas may be classified into the same structural group because they are infrequently shown simultaneously. Note that the target structures here include, for example, inferred structures inferred by an inference model that analyzes surgical images acquired by a camera.

[0057] Furthermore, the drawing colors used to depict the target structure are classified into two or more color groups with consecutive hue angles. Note that if the target structure is an inferred structure, the drawing color may be called the inferred display color.

[0058] Here, for the colors belonging to each color group, colors that can be distinguished as different colors from each other within each color group are assigned to that group. On the other hand, within each color group, assignments may be made without regard to the color difference condition.

[0059] After classifying and creating color groups as described above, a different color group is assigned to each structure group. In other words, each structure group is assigned a different color group, or it is possible to select a different color group for each structure group.

[0060] Furthermore, if a drawing color is selected for a target structural element belonging to a certain structural group, it is possible to prevent the selection of colors similar to the selected drawing color for target structural elements belonging to different structural groups.

[0061] Hereinafter, this embodiment will be described in the order of prior art and the technology of this embodiment. (Conventional technologies and their challenges) The display color of the inference results must be selected from colors that do not overlap with those of the surgical field. However, the color tones of the surgical field are biased. Therefore, the range of colors that can be selected as the display color of the inference results is limited. As a result, even if different colors are assigned to multiple structures and displayed simultaneously, it may not be possible to ensure sufficient distance between the colors, resulting in them being displayed in similar colors. When colors are too similar, there is a risk that the surgeon may misidentify the structures.

[0062] Therefore, in this embodiment, image colors are assigned as follows. First, multiple structures are divided into structures that are rarely used simultaneously or structures that are unlikely to be used simultaneously. These divided groups of structures are called, for example, the first structure group, the second structure group, and the third structure group. In other words, structures are classified into structure groups.

[0063] Structures that are unlikely to be displayed simultaneously belong to the same structure group. Note that the number of structure groups is not limited to three; for example, there may be four or more structure groups.

[0064] Display colors, or drawing colors, are classified into at least three consecutive color groups according to their hue angle, excluding colors that frequently appear in the surgical field. Then, each structural group is assigned a separate color group. Alternatively, it may be possible to select a different color group for each structural group.

[0065] In classifying color groups, we ensure that there is a color difference between the classified color groups that allows them to be distinguished as different colors. In other words, colors belonging to different color groups must have a distinguishable color difference. Color differences within the same color group are not considered.

[0066] This prevents color misidentification between different structural groups. Furthermore, within a structural group, more display colors can be assigned to each structure. As a result, the degree of freedom in selecting drawing colors is increased.

[0067] Figure 7 shows an example of image color assignment in this embodiment. Figure 7 is a diagram of the HSV (Hue, Saturation, Value) space, showing an example where the drawing color is divided into cyan, green, and other colors.

[0068] In the example shown in Figure 7, the color groups are classified into cyan-based color groups, green-based color groups, and other colors. Furthermore, these classifications exclude red-based colors, which are frequently seen in the surgical field.

[0069] Figure 7 plots a portion of the drawing colors from drawing color 0 (D0) to drawing color 17 (D17) as display colors, i.e., drawing colors. In each plot, the large circles indicate 0% transparency. In each plot, the lines and small circles indicate the change in position in HSV space in response to changes in transparency. In the example shown in Figure 7, the settings are configured to ensure color differences between the cyan color group, the green color group, and the other color groups that are displayed simultaneously.

[0070] In the example shown in Figure 7, color differences are maintained between drawing colors 0 (D0), 1 (D1), and 17 (D17), which belong to the cyan color group; drawing colors 2 (D2), 3 (D3), 4 (D4), 5 (D5), and 7 (D7), which belong to the green color group; and drawing color 6 (D6), which belongs to the other color group. As shown in Figure 7, the color differences between each drawing color decrease as the transparency increases. This is because all plots are approaching the origin of the HSV space.

[0071] In this embodiment, it is preferable that the drawing colors belonging to different color groups are set so that a predetermined color difference is maintained even when the transparency increases and they approach the origin of the HSV space.

[0072] With this setting, if a drawing color is selected for a target structural part included in one structural group, it is possible to create a state where, for target structural parts included in different structural groups, colors close to the selected drawing color are not selected as the drawing color.

[0073] The surgical image processing apparatus 1 of Embodiment 2 will now be described. The surgical image processing apparatus 1 of Embodiment 2 can be the same as the surgical image processing apparatus 1 of Embodiment 1. In Embodiment 2, the drawing color determination unit 14 can perform the main parts of the surgical image processing method of Embodiment 2 described above. For example, the creation of structure groups, the creation of color groups, and the assignment of color groups to structure groups, which relate to the assignment of image colors, can be performed by the drawing color determination unit 14.

[0074] Note that the configuration of the surgical image processing device 1 in Embodiment 2 does not have to be the same as the configuration of the surgical image processing device 1 in Embodiment 1. For example, it may include a part other than the drawing color determination unit 14 that performs a part of the surgical image processing method of Embodiment 2, such as creating a structural group.

[0075] (Embodiment 3) The surgical image processing method and surgical image processing apparatus 50 of Embodiment 3 will be described with reference to Figures 8 and 9. The following description will focus on aspects that differ from Embodiment 1 or Embodiment 2. Matters not specifically described can be the same as in Embodiment 1 or Embodiment 2. Embodiment 1 mainly concerned the selection of image colors, and Embodiment 2 mainly concerned the assignment of image colors. In contrast, Embodiment 3 concerns the automatic adjustment of drawing colors.

[0076] The outline of the surgical image processing method of Embodiment 3 will be described. For example, when rendering an image in which the analysis results of an inference model that analyzes a surgical image acquired by a camera are reflected in the surgical image, when generating the rendering color of the inference result, a background color is generated from the entire image, the region including the rendering of the inference result, or the pixels surrounding the pixels in which the inference result is rendered. The color distance between the rendering color and the background color is calculated as a color difference delta E in a color space with color difference uniformity. It is determined whether the calculated color difference delta E is greater than or less than a preset threshold, and based on the determination result, the rendering color is corrected so that the color difference delta E is larger or smaller. Alternatively, information on the color difference delta E or warning information may be output.

[0077] By correcting the drawing color as described above, the drawing color can be changed so that it is not mistaken for the surrounding colors.

[0078] Furthermore, when correcting the drawing color as described above, the change in color difference delta E may be limited to a range that can be recognized as the same color, and this limiting range may be shifted over time.

[0079] By correcting the drawing color as described above, it becomes possible to prioritize the relationship between the target structure and the drawing color, or to gradually change the drawing color within the constraint that the target structure should be displayed with a specific drawing color. Furthermore, by shifting the constraint range over time, expanding the constraint range over time, or increasing the amount of change over time, it becomes easier to make the drawing color appear the same before and after the change.

[0080] Hereinafter, this embodiment will be described in the order of prior art and the technology of this embodiment. (Conventional technologies and their challenges) For example, even if colors are used that allow for distinction between the drawing color and the background color, or between different drawing colors, situations may arise where the drawing color and the background color, or different drawing colors, become close together, depending on the settings of the endoscope used for imaging, or the environment in which the endoscope is used. This is because colors can change depending on the settings of the endoscope or the environment in which the endoscope is used. Furthermore, when processing images acquired with different endoscopes together, or when processing images acquired with one endoscope using another endoscope, it may become difficult to distinguish between colors.

[0081] Therefore, the surgical image processing device 50 of this embodiment analyzes the drawing color and the background color in real time. Then, it automatically changes at least one of the hue, saturation, and brightness of the drawing color according to the endoscopic image so that the drawing color maintains a certain color distance from the background color, and then displays the image.

[0082] When changing at least one of the hue, saturation, and brightness, the changes should be made gradually within a range where the drawn color is still perceived as having the same color tendency. This allows the drawn color to be changed while appearing to be the same color before and after the change.

[0083] An example of the surgical image processing method of Embodiment 3 will be described with reference to Figure 8. Figure 8 is a diagram showing the HSV space, which includes the distribution of each drawing color and the color of the surgical field. In Figure 8, the drawing color E and the distribution of colors C121 of the surgical field sampled from the surgical image are shown in the HSV space. In Figure 8, E3 represents the drawing color 3. The same applies to the other drawing colors.

[0084] The range E3A shown in Figure 8 indicates the range in which the drawing color 3 (E3) changes. While range E3A exemplifies the range in which the drawing color 3 (E3) changes, the drawing color changes similarly for other drawing colors as well. The drawing color changes based on factors such as confidence level and transparency.

[0085] The double-headed arrows M in Figure 8 indicate the distance between the drawing color and the color of the surgical field. The color of the surgical field can also be called the background color. The double-headed arrow M1 in Figure 8 indicates the closest distance between the drawing color 7 (E7) and the color distribution C121 of the surgical field. Similarly, the double-headed arrow M2 in Figure 8 indicates the closest distance between the drawing color 17 (E17) and the color distribution C121 of the surgical field. Furthermore, the double-headed arrow M3 in Figure 8 indicates the closest distance between the drawing color 10 (E10) and the color distribution C121 of the surgical field.

[0086] The double-headed arrows N in Figure 8 indicate the distance between drawing colors. Double-headed arrow N1 in Figure 8 indicates the closest distance between drawing color 14 (E14) and drawing color 12 (E12). Similarly, double-headed arrow N2 in Figure 8 indicates the closest distance between drawing color 12 (E12) and drawing color 10 (E10).

[0087] In this embodiment, when automatically changing at least one of the hue, saturation, and brightness of each drawing color according to the endoscopic image, the range of change for each drawing color is such that the distance between the drawing color and the color of the surgical field, and the distance between that drawing color and other drawing colors, are at a distance that allows for color discrimination. In other words, the distance M between the drawing color and the color of the surgical field, and the distance N between drawing colors are set to be greater than or equal to a predetermined value.

[0088] Referring to Figure 9, the surgical image processing device 50 of this embodiment will be described. This is a block diagram showing the configuration of the surgical image processing device 50 of this embodiment. As an example of a process for automatically changing the drawing color, a process can be considered in which the drawing color and background color are calculated from the input image and the inference result, and a correction amount that ensures the color distance is found is found and reflected in the calculation of the drawing color. The color distance refers to the distance between the drawing color and the color of the surgical field, and the distance between drawing colors.

[0089] An example configuration of the surgical image processing device 50 that performs the above-described processing will now be explained. As shown in Figure 9, the surgical image processing device 50 includes an inference unit 52, a background color analysis unit 54, a base color analysis unit 56, a drawing color correction amount calculation unit 60, and a drawing color calculation unit 62.

[0090] The inference unit 52 is the part that analyzes surgical images acquired by a camera such as an endoscope. The surgical images acquired by the camera are input to the inference unit 52. The inference unit 52 also stores an inference model used for analyzing the surgical images.

[0091] The background color analysis unit 54 is the part that generates the background color by analyzing the image. The background color analysis unit 54 receives input such as the surgical image acquired by the camera and information about the target structure inferred by the inference unit 52.

[0092] The base color analysis unit 56 is the part that generates the base color by analyzing the image, etc. Information about the target structure inferred by the inference unit 52 is input to the base color analysis unit 56.

[0093] The drawing color correction amount calculation unit 60 is responsible for calculating the correction amount when correcting the drawing color. The drawing color correction amount calculation unit 60 calculates the amount to change the hue, saturation, and brightness, etc. Parameters that are referenced when calculating the drawing color correction, such as the settings of the endoscope or the usage environment of the endoscope, may be input to the drawing color correction amount calculation unit 60. In addition, the drawing color correction amount calculation unit 60 may change the correction amount calculated over time.

[0094] The drawing color calculation unit 62 is the part that generates the drawing color. The drawing color calculation unit 62 generates the drawing color by referring to the correction amount calculated by the drawing color correction amount calculation unit 60. When generating the drawing color, the drawing color calculation unit 62 generates the drawing color so that the color difference between the drawing color and the background color, and the color difference between drawing colors, are values ​​of a desired value or a desired range. The image determined by the above processing is then transmitted to the drawing unit or the like as an image in which, for example, the analysis results by the inference model are reflected in the surgical image, and is drawn and displayed.

[0095] The above explains how to calculate the correction amount for the drawing color based on the result of comparing the color difference with a predetermined value. However, if the difference between the color difference and the predetermined value is not within the desired range, the correction amount may not be calculated, or a warning or control information may be output along with the calculation of the correction amount. Here, warning information includes information such as warnings about colors that are difficult to recognize and suggestions for easily recognizable colors (specified colors). Control information includes information to forcibly stop surgical image processing with colors that are difficult to recognize and information to prioritize the display of easily recognizable colors.

[0096] The surgical image processing device 1 may be configured to automatically change a pre-specified color according to the background color and surgical scene. In this case, the recognition of the background structure and surgical scene is performed by an AI model that recognizes them. Depending on the background color and surgical scene, the device can automatically display a pre-specified color that has been confirmed to satisfy the desired color difference, either in advance or in real time. This makes it possible to display drawing colors that ensure appropriate color differences depending on different surgical scenes (organ colors differ depending on the surgical procedure) and surgical conditions (e.g., heavy bleeding). The device may also have a function to adjust transparency without changing the specified color.

[0097] Furthermore, the system can adjust the hue, saturation, brightness, and transparency of the specified or drawn color according to the user. Multiple pieces of information, such as the user's chosen color and ease of recognition, are input, and the system analyzes the user's tendencies. By adjusting the color balance according to the user's preferences, a more visually appealing display becomes possible.

[0098] If the surgical image processing device 1 is configured to change a specified color according to the background color or the like, the surgical image processing method may further include a step to adjust the specified color. Also, if the surgical image processing device 1 is configured to change a specified color according to the background color or the like, the surgical image processing device may further include a color adjustment unit to adjust the specified color.

[0099] If the surgical image processing device 1 is capable of adjusting a specified color or drawing color according to the surgeon, the surgical image processing method may further include a step in which a value determined by the surgeon for adjusting at least one of the specified color and the drawing color is input. Also, if the surgical image processing device 1 is capable of adjusting a specified color or drawing color according to the surgeon, the surgical image processing device may further include an input unit into which a value determined by the surgeon for adjusting at least one of the specified color and the drawing color is input.

[0100] As described above, the surgical image processing device 1 allows for the modification of a specified color depending on the surgical scene, background, and surgeon, or, without changing the specified color itself, to fine-tune the saturation, brightness, transparency, etc. This allows for adjustment of the drawing color to make it easier for the surgeon to see. In this case, a predetermined color difference is ensured from the standpoint of color identification. However, the desired color difference can be set to various values ​​depending on the surgical scene and the surgeon's preferences.

[0101] For example, the structures that appear in the background (surgical field) differ between gastric surgery and lung surgery. Different adjacent organs in the surgical field can subtly alter which colors are easiest to see. For instance, a certain color might be easily visible during gastric surgery, while it might be difficult to see during lung surgery. The surgical image processing device 1 described above can automatically change pre-specified colors according to the background color and surgical scene. Therefore, it can provide an easily viewable display without requiring any special settings.

[0102] The automatic change of the specified color described above can also be performed using an AI model, for example. One example is as follows: Suppose an AI model for recognizing loose connective tissue is being used. In this state, the surgical scene and background structure are recognized by another AI model. This other AI model determines the color of the surgical field and decides on a color that makes the loose connective tissue more visible. The surgical image processing device 1 can then automatically adjust the drawing color and display information based on the determined color.

[0103] Furthermore, the drawing color may be automatically adjusted according to the surgical situation. For example, in situations with heavy bleeding, the surgical field will appear more red. In situations with strong redness, red tones may be difficult to see. Therefore, the drawing color may be automatically adjusted to avoid red tones.

[0104] Furthermore, color vision varies from surgeon to surgeon. Each surgeon has their own preferences for colors that are easy to see and colors they like. Therefore, when adjusting the specified color or drawing color according to the surgeon, the surgical image processing device 1 can analyze the surgeon's color preferences and make adjustments based on the results. Alternatively, the surgical image processing device 1 may be configured to accept information about the surgeon's preferences in advance. In this case, the surgical image processing device 1 can automatically adjust the drawing color or specified color by referring to the input information.

[0105] The embodiments of the present invention have been described above. The present invention is not limited to the embodiments described above, and various modifications, variations, and combinations are possible.

[0106] For example, if the inference unit infers an object as having multiple structures, the drawing color of that object may be set to a different color from the drawing colors of the other objects. In other words, the area where the inference unit has made redundant inferences, such as double inference, is highlighted by changing its drawing color. This increases the attention of the redundantly inferred area, draws the operator's attention to it, and prompts them to make a judgment about what the correct structure is. This is because if the AI ​​is unsure of its judgment, the operator is likely to be unsure as well.

[0107] Furthermore, areas with small color differences between displayed parts may be highlighted, especially in cases where double inference occurs. This can encourage the operator to observe more carefully.

[0108] Furthermore, the inference unit may highlight areas where the inference is below a predetermined level of confidence. This can help prevent the operator from making incorrect judgments. [Explanation of Symbols]

[0109] 1. Surgical image processing device 10 Anatomical structure recognition section 12. Base color determination section 14 Drawing Color Determination Unit 16 Background color determination section 20 Color difference calculation section 22 Color difference judgment section 30 Reasoning section 32 Drawing section 50 Surgical Image Processing Devices 52 Reasoning part 54 Background color analysis section 56 Base color analysis section 60 Drawing color correction amount calculation unit 62 Drawing color calculation section

Claims

1. A surgical image processing method, The steps include determining the drawing color, which is the display color of the target structure, The steps include determining the background color, which is the display color of the pixels or surrounding area of ​​the target structure, A step of calculating the color difference between the drawing color and the background color, The step includes determining whether the color difference exceeds a predetermined value. Surgical image processing methods.

2. In the step of determining the background color, a plurality of background colors are determined according to the base color, which is the original color of the target structure. In the step of determining the color difference, the color difference between the drawing color and the plurality of background colors is determined. The surgical image processing method according to claim 1.

3. In the step of determining the drawing color, the drawing color is determined by referring to a pre-specified designated color. The further step includes adjusting the specified color based on at least one of the surgical scene and background included in the surgical image. The surgical image processing method according to claim 1 or 2.

4. In the step of determining the drawing color, the drawing color is determined by referring to a pre-specified designated color. The step further includes inputting a value determined by the operator for adjusting at least one of the specified color and the drawing color, The surgical image processing method according to claim 1 or 2.

5. A surgical image processing device, A drawing color determination unit that determines the drawing color, which is the display color of the target structure, A background color determination unit determines the background color which is the display color of the pixels or surrounding area of ​​the target structure, A color difference calculation unit that calculates the color difference between the drawing color and the background color, The system includes a color difference determination unit that determines whether the color difference exceeds a predetermined value. Surgical image processing device.

6. The drawing color determination unit determines the drawing color by referring to a pre-specified designated color. The color adjustment unit further includes a color adjustment unit that adjusts the specified color based on at least one of the surgical scene and background included in the surgical image. The surgical image processing apparatus according to claim 5.

7. The drawing color determination unit determines the drawing color by referring to a pre-specified designated color. It further includes an input section into which a value is input for adjusting at least one of the specified color and the drawing color, determined by the operator. The surgical image processing apparatus according to claim 5.

8. A surgical image processing method, The steps include classifying the target structure into two or more groups of structures that are used infrequently or are unlikely to be used simultaneously, The steps include dividing the colors used to display the target structure into two or more color groups with consecutive hue angles, wherein there is a color difference between colors belonging to different color groups that allows them to be distinguished as different colors, The step of assigning each of the aforementioned structural groups a color from a different color group to be used as a drawing color which is the display color of the target structure, Surgical image processing methods.

9. A surgical image processing method, Steps include recognizing the target structure, The step includes determining the drawing color of the target structure, In the step of determining the drawing color, if the target structure belongs to a different structure group, the drawing color is determined such that there is a color difference exceeding a predetermined value between each of the drawing colors. Surgical image processing methods.

10. In the step of recognizing the target structure, two or more of the target structures are recognized, In the step of determining the drawing color of the target structure, the drawing color is determined such that there is a color difference exceeding a predetermined value between the drawing colors of different target structures. The surgical image processing method according to claim 9.

11. The steps include classifying the aforementioned target structures into two or more groups of structures that are used infrequently or are unlikely to be used simultaneously, The steps include classifying the colors used to display the target structure into two or more color groups with consecutive hue angles, wherein there is a color difference between colors belonging to different color groups that allows them to be distinguished as different colors, The further step includes assigning each of the aforementioned structural groups a color from a different color group to be used as a drawing color which is the display color of the target structure. In the step of determining the drawing color, the drawing color is determined such that there is a color difference exceeding a predetermined value between the drawing colors of two target structures belonging to different groups of structures. The surgical image processing method according to claim 9 or 10.

12. A surgical image processing device, A function to classify target structures into two or more structural groups that are infrequently used simultaneously or are unlikely to be used simultaneously. A function to divide the colors used to display the target structure into two or more color groups with consecutive hue angles, wherein there is a color difference between colors belonging to different color groups that allows them to be distinguished as different colors. Each of the aforementioned structure groups is provided with a drawing color determination unit having the function of assigning a color from a different color group to be used as a drawing color, which is the display color of the target structure. Surgical image processing device.

13. A surgical image processing method, A step of calculating the color difference between the drawing color, which is the display color of the target structure, and the background color, which is the display color of the pixels or surrounding area of ​​the target structure. A step of comparing the aforementioned color difference with a predetermined value, The step of correcting the drawing color based on the result of the comparison between the color difference and the predetermined value, Surgical image processing methods.

14. In the step of correcting the drawing color, the content of the correction is limited to a range in which the color can be recognized as the same before and after the correction, and this limited range changes over time. The surgical image processing method according to claim 13.

15. A surgical image processing device, A function to calculate the color difference between the drawing color, which is the display color of the target structure, and the background color, which is the display color of the pixels or surrounding areas of the target structure. A function to compare the aforementioned color difference with a predetermined value, The system includes a drawing color correction amount calculation unit that has the function of calculating the drawing color correction amount based on the result of the comparison between the color difference and the predetermined value, Surgical image processing device.

16. A surgical image processing method, A step of calculating the color difference between the drawing color, which is the display color of the target structure, and the background color, which is the display color of the pixels or surrounding area of ​​the target structure. A step of comparing the aforementioned color difference with a predetermined value, The step of outputting warning information or control information based on the result of the comparison between the color difference and the predetermined value, Surgical image processing methods.

Images