Surgical imaging equipment

Abstract

A surgical image processing device 1 is provided that converts the image quality of a captured image into a desired image quality. The surgical image processing device 1 comprises a preprocessing unit 10 including an image quality converter 20 that converts the image quality of an output image of a surgical image acquired by a camera 102, an inference unit 12 including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image, and a drawing unit 14 that draws an image in which the analysis result by the inference model is reflected in the surgical image, and the image quality converter 20 comprises an inverse gamma processing means that converts the output image into a linear signal, a means that performs image quality conversion on the linear signal, including at least one of color conversion, luminance conversion, noise reduction, and edge emphasis, and a gamma processing means that performs gamma processing on the converted image.

Description

[Technical field]

[0001] The present invention relates to a surgical image processing apparatus. [Background technology]

[0002] 2. Description of the Related Art There is known a technique for assisting surgery by acquiring surgical images obtained by photographing a surgery performed by a surgeon, analyzing the surgical images, and applying the analysis results to the surgical images.

[0003] Patent Document 1 describes that when processing an endoscopic image captured by an endoscopic device, the image quality is converted based on an image quality adjustment value set in the endoscopic device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7130038 Summary of the Invention [Problem to be solved by the invention]

[0005] When converting the image quality of an endoscopic image captured by an endoscopic device to a predetermined image quality, the image quality adjustment value of the endoscopic device that captured the endoscopic image may not be clear. In addition, the gradation response characteristic of the image signal of the endoscopic image output from the endoscopic device is generally nonlinear. In such a case, no technology has been proposed for converting the image quality of the captured image to a desired image quality.

[0006] An object of the present invention is to provide a surgical image processing device that converts the image quality of a captured image into a desired image quality. [Means for solving the problem]

[0007] The surgical image processing device of the present invention is a surgical image processing device included in a surgical support system, and comprises a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera, an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image, and a drawing unit that draws an image in which the analysis results by the inference model are reflected in the surgical image, and the image quality converter comprises an inverse gamma processing means that converts the output image into a linear signal, a means for performing image quality conversion on the linear signal including at least one of color conversion, brightness conversion, noise reduction, and edge enhancement, and a gamma processing means that performs gamma processing on the converted image. Effect of the Invention

[0008] According to the present invention, it is possible to provide a surgical image processing device that converts the image quality of a captured image into a desired image quality. [Brief description of the drawings]

[0009] [Figure 1A] 1 is a diagram showing an outline of the functional configuration of a surgery support system to which a surgery image processing device of this embodiment is applied. [Figure 1B] FIG. 13 is a diagram showing a modified example of the image quality converter 20. [Diagram 2] FIG. 1 is a diagram showing an overview of conventional image quality conversion. [Diagram 3] FIG. 1 is a diagram showing an overview of image quality conversion according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing a processing flow in a surgery support system. [Diagram 5] FIG. 1 is a diagram illustrating a conventional method for estimating inverse gamma. [Figure 6] FIG. 4 is a diagram illustrating a method of estimating an inverse gamma according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing a white balance checking tool according to the present embodiment. [Figure 8] FIG. 13 is a diagram showing a state in which the white balance checking jig is separated. [Figure 9] 13A and 13B are diagrams illustrating a white balance checking tool according to another embodiment. [Figure 10] FIG. 1 is a diagram showing an example of conventional exposure adjustment. [Figure 11] 6A and 6B are diagrams illustrating how to determine a fixed exposure correction ratio. [Figure 12] FIG. 11 is a diagram illustrating an example of exposure correction. [Figure 13] FIG. 1 is a diagram illustrating selection of a conventional image quality conversion formula. [Figure 14] 11A and 11B are diagrams illustrating selection of an image quality conversion formula in the present embodiment. [Figure 15] FIG. 1 is a diagram illustrating an example of conventional image quality conversion. [Figure 16] FIG. 1 shows steps of conventional conversion from image quality A-3 to image quality B-2. [Figure 17] 1A to 1C are diagrams illustrating an example of image quality conversion according to the present embodiment. [Figure 18] FIG. 13 is a diagram showing the steps of conversion from image quality A-3 to image quality B-2 in this embodiment. [Figure 19] FIG. 1 is a diagram showing an example of conventional image quality conversion in the case where three endoscopes are used. [Figure 20] 11A to 11C are diagrams illustrating an example of image quality conversion according to the present embodiment when three endoscopes are used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the accompanying drawings, in which: Fig. 1A is a diagram showing an outline of the functional configuration of a surgery assistance system 100. The surgery assistance system 100 is provided with a surgery image processing device 1.

[0011] (Surgery support system) The surgery support system 100 is a system that supports a surgeon in surgery. Images captured during a surgeon's surgery are called surgery images. The surgery images are subjected to various processes in the surgery support system 100. The surgery support system 100 supports the surgeon in surgery by displaying the processed images.

[0012] The surgery support system 100 includes a camera 102, a display device 104, and a surgery image processing device 1. The camera 102 is a device that acquires a surgery image. An example of the camera 102 is an endoscope camera. The camera 102 is not limited to an endoscope camera as long as it is a camera that captures the state of surgery. The camera 102 may be, for example, a camera installed in an operating room or a camera attached to a surgery support robot. The camera 102 may also be a camera that uses invisible light, hyperspectral imaging, etc. in addition to visible light. The display device 104 is a device that displays an output image from the display unit 16. Examples of the display device are a console and an LCD.

[0013] (Surgery image processing device) The surgical image processing device 1 is a device that determines the display contents of the surgery support system 100. The surgical image processing device 1 includes a pre-processing unit 10, an inference unit 12, and a drawing unit 14. The surgical image processing device 1 may also include a display unit 16. The pre-processing unit 10, the inference unit 12, the drawing unit 14, and the display unit 16 process the surgical image captured by the camera 102.

[0014] (Inference part) An AI model that analyzes surgical images and infers anatomical structures, instrument trajectories, events such as bleeding, surgical procedures and scenes, and the state of instruments relative to anatomical structures is called an inference model. The inference unit 12 includes an inference model that is applied to the surgery support system 100.

[0015] (Drawing section) The surgical image on which the analysis result of the inference unit 12 is reflected is called a drawn image. The manner in which the analysis result of the inference unit 12 is reflected in the drawn image is called a drawing manner. The drawing unit 14 draws the drawn image in accordance with the determined drawing manner.

[0016] (Display) The manner in which a drawn image is displayed on a display device is called a display manner. The display unit 16 displays the drawn image in accordance with a determined display manner. The drawn image displayed by the display unit 16 may be an image whose image quality has been converted by the image quality converter 20 of the pre-processing unit 10 described below, or may be an image before the image quality is converted. In the latter case, the inference unit 12 performs inference using the image whose image quality has been converted, and when the image is subsequently displayed on the display unit 16, the image before the image quality is converted is used. The display unit 16 may be a display separate from the surgical image processing device 1, or may be displayed on a display attached to the display unit 16 or the surgical support robot.

[0017] (Preprocessing section) The surgical image processing device 1 further includes a pre-processing unit 10. The pre-processing unit 10 will be described below.

[0018] The surgical images are acquired by a camera 102 in a medical institution or the like. Information indicating the state of the body is called body information. Information indicating the state of the instruments operated by the surgeon is called instrument information. In the inference unit 12, the body information and instrument information contained in the surgical images are input to an inference model. In the inference model, the body information and instrument information are analyzed by AI. The inference model infers the anatomical structure, the trajectory of the instrument, and the state of the instrument relative to the anatomical structure, etc.

[0019] The manufacturers of the cameras 102 used in medical institutions are various. The models of the cameras 102 used in medical institutions are various. The image quality of the acquired surgical images differs depending on the model of the camera 102. Examples of the image quality include spatial frequency, brightness, and color tone.

[0020] The inference model is generated by learning surgical images. The surgical images that the inference model learns from are called training images. The training images are acquired by a specific camera 102.

[0021] The surgical images used in the surgical image processing device 1 are called images for inference. More specifically, the surgical images input to the inference model are called images for inference. The camera that captured the learning images may be different from the camera 102 that captured the images for inference.

[0022] In this case, the image quality of the training image and the image quality of the image for inference may differ. If the image quality of the training image and the image quality of the image for inference differ, the accuracy of the analysis result in the inference unit 12 may decrease.

[0023] The pre-processing unit 10 is a unit that reduces the difference between the image quality of the training image and the image quality of the image for inference. The pre-processing unit 10 suppresses a decrease in the accuracy of the analysis result of the inference unit 12 by reducing the difference between the image quality of the training image and the image quality of the image for inference.

[0024] The pre-processing unit 10 converts the image quality of the image for inference into the image quality of the learning image. The conversion is performed by a conversion formula. The conversion formula is a formula for converting the image quality of the image for inference into an image quality that is close to the image quality of the learning image. In addition, the learning image may be converted into the image for inference, and the subject of the image does not matter as long as the method reduces the image quality difference.

[0025] The conversion formula is set in the pre-processing unit 10. A person who uses the surgery support system 100 is called a user. The conversion formula is set by the user. Alternatively, the conversion formula may be set automatically.

[0026] The image transformed in the preprocessing unit 10 is called a transformed image. The transformed image is input to the inference unit 12.

[0027] The conversion formula may be a formula for converting the image quality of the image for inference into an image quality that improves the analysis accuracy of the inference unit 12. The content of the conversion formula is not limited. The conversion formula may be a formula for performing frequency analysis on the image for inference and converting the image by a frequency approach.

[0028] The database in which the conversion formula is stored is called the image quality converter 20. The image quality converter 20 is included in the pre-processing unit 10. The image quality converter 20 may be stored in a storage means of the surgery support system 100. The image quality converter 20 may be downloaded from the cloud, a server, or the like.

[0029] An overview of image quality conversion will be described below. Fig. 2 is a diagram showing an overview of conventional image quality conversion. Fig. 3 is a diagram showing an overview of image quality conversion according to this embodiment. (Output image) The image output by the camera 102 is called the output image 110. The image captured by the camera 102 is usually subjected to gamma processing before being output from the camera 102. When the image is subjected to gamma processing, the tone response characteristic of the image becomes nonlinear. The tone response characteristic of the output image 110 is usually nonlinear.

[0030] Hereinafter, when the tone response characteristic of an image is nonlinear, it will be said that the image signal is nonlinear, etc. Similarly, when the tone response characteristic of an image is linear, it will be said that the image signal is linear, etc.

[0031] (Image quality conversion) As shown in FIG. 2, when the image signal of the output image 110 is nonlinear, image quality conversion is performed on the nonlinear signal.

[0032] An example will be described in which the image quality conversion is color conversion. The image after the image quality conversion is called a converted image 112. The color conversion is performed by performing a matrix operation on each of the R, G, and B signals. The matrix operation is performed by a conversion matrix. Alternatively, the color conversion is performed by adjusting the intensity of each of the R, G, and B signals.

[0033] When a nonlinear signal is color-converted, the color is converted to a different color depending on the brightness. As a result, the image quality of the output image cannot be converted to an image quality that is close to the image quality of the learning image. The difference between the image quality of the converted image 112 and the image quality of the learning image becomes large.

[0034] Conversion of image brightness is similar to conversion of image color. Brightness correction is performed by multiplying the image signal by a gain. When converting a nonlinear signal, the weight of the gain differs depending on the brightness. As a result, the image quality of the output image 110 cannot be converted to an image quality that is close to the image quality of the learning image. As with color and brightness, when a nonlinear signal is converted, the contrast and sharpness also do not produce the intended image quality conversion results.

[0035] In the embodiment, as shown in Fig. 3, inverse gamma processing is performed on the output image 110. The nonlinear signal of the output image 110 is converted into a linear signal by the inverse gamma processing. When the output image 110 becomes a linear signal, the linear signal is subjected to image quality conversion to obtain an intended image quality conversion result.

[0036] Next, the color-converted image is subjected to gamma processing equivalent to that of the learning image, whereby the output image 110 is converted into a converted image 112 having the image quality of the learning image.

[0037] By converting the linear signal into an image, even if the brightness is different, it is possible to obtain image quality equivalent to that of the camera to which the signal is converted, for example, the camera that captured the surgical image on which the inference model was trained.

[0038] The flow of processing in the surgery support system 100 will be described in more detail with reference to Fig. 4. Fig. 4 is a diagram showing the flow of processing in the surgery support system 100. The following description will focus on the processing in the image quality converter 20.

[0039] (Source camera) The camera 102 is the camera from which the conversion is performed. An example of the camera 102 is an endoscope camera. Signal processing in the camera 102 will be described. The camera 102 includes an image sensor IS. The image sensor IS acquires a surgical image. The image signal of the image acquired by the image sensor IS is a linear signal. The image signal is configured in a Bayer array of R, Gr, B, and Gb.

[0040] (Linear processing (first stage)) The image signal acquired by the Image Sensor IS undergoes a series of signal processing steps included in Linear Processing (front stage) 107. Linear Processing (front stage) 107 includes processes A-1 to A-5 and C-1. The composition of the image signal is converted by undergoing each process in Linear Processing (front stage) 107. The composition of the image signal changes from Bayer array data of R, Gr, B, Gb to data separated into R, G, B.

[0041] The linear processing (front stage) 107 includes BLC (Black Level Correction) A-1, Sensor correction (Defect pixel / Shading / Linearity) A-2, Lens correction (Shading / Magnification chromatic aberration) A-3, RAW-NR (Noise reduction) A-4, White Balance A-5, and DeMosaic C-1. The image signal acquired by the Image Sensor IS undergoes the above-mentioned processes in the above-mentioned order.

[0042] (Linear processing (second stage)) The image signal converted into data separated into R, G, and B in linear processing (previous stage) 107 is further processed in linear processing (next stage) 108. In linear processing (next stage) 108, the image signal is converted from a linear signal to a nonlinear signal.

[0043] Linear processing (back stage) 108 includes a CCM (Color Correction Matrix) color conversion A-6 and a gamma (gamma) process C-2. The image signal converted into data separated into R, G, and B undergoes the above-mentioned processes in the above-mentioned order. The image signal becomes a nonlinear signal by undergoing the gamma process C-2.

[0044] (Non-Linear processing) The image signal converted into a non-linear signal in Linear processing (later stage) 108 is further processed in Non-Linear processing (non-linear processing) 109. The image signal is converted from the RGB format to the YCbCr format in Non-Linear processing 109. The image signal that has been subjected to Non-Linear processing 109 is output from camera 102.

[0045] The non-linear processing 109 includes RGB-NR (noise reduction) A-8, RGB2YC (RGB to YCbCr) C-3, luminance correction and color difference correction A-9, edge emphasis A-10, and YC-NR (noise reduction) A-11. The non-linear signal of the RGB format undergoes the above-mentioned processes in the above-mentioned order.

[0046] The image that has undergone the above processes is output from the camera 102 as an output image. The signal output from the camera 102 is called an output signal. The nonlinear signal of the RGB format is converted into a nonlinear signal of the YCbCr format by the RGB2YC processing C-3. The image signal of the output image is a nonlinear signal of the YCbCr format.

[0047] (Image quality converter) The following describes the processing in the image quality converter 20. The processing in the image quality converter 20 includes processing 21 for converting the output signal, which is a nonlinear signal, into a linear signal, and processing 22 for converting the image quality of the output image into that of the destination camera. The conversion processing 21 into a linear signal is performed by the inverse operation of the processing of the source camera.

[0048] (Conversion to linear signal) The conversion process 21 to a linear signal will be described. The conversion process 21 to a linear signal includes YC2RGB (YCbCr to RGB) C-3', inverse edge emphasis A-10', inverse luminance correction and inverse color difference correction A-9', and γ -1 and an inverse gamma C-2' (inverse gamma processing means). These processes are the inverse operations of the corresponding processes in the linear processing (subsequent stage) 108 and the non-linear processing 109. The signs of the processes in the image quality converter 20 are the signs of the corresponding processes in the linear processing (subsequent stage) 108 and the non-linear processing 109 with (') added.

[0049] The YCbCr nonlinear signal input to the image quality converter 20 is converted into an RGB linear signal in a linear signal conversion process 21. The RGB linear signal is subjected to offset correction E-1 (offset adjustment), exposure correction (gain correction) E-2 (brightness adjustment), and white balance correction E-3 (white balance correction).

[0050] The image signal that has been processed from E-1 to E-3 undergoes an inverse CCM color conversion A-6'. The inverse CCM color conversion A-6' is the inverse operation of the CCM color conversion A-6 in the linear processing (previous stage) 107. The inverse CCM color conversion A-6' can be omitted.

[0051] (Conversion process to the image quality of the destination camera) The conversion process 22 to convert image quality into that of the destination camera will now be described. The conversion process 22 to convert image quality into that of the destination camera is similar to the processes included in the Linear processing (later stage) 108 and the Non-Linear processing 109.

[0052] The conversion process 22 to the image quality of the destination camera includes CCM color conversion B-6, γ (Gamma) processing D-2 (gamma processing means), RGB-NR B-8 (noise reduction), RGB2YC D-3, brightness correction and color difference correction B-9 (color conversion, brightness conversion), edge emphasis B-10, and YC-NR B- 11. The image signal undergoes the above processes in the above order.

[0053] The purpose of CCM color conversion B-6 in conversion process 22 to the image quality of the destination camera is similar to the purpose of CCM color conversion A-6 in Linear processing (later stage) 108. However, CCM color conversion B-6 has conversion contents corresponding to the destination camera.

[0054] As for other processes, the purpose of processes with the same names is similar between processes included in conversion process 22 to the image quality of the destination camera and processes included in Linear processing (later stage) 108 and Non-Linear processing 109. However, the specific contents of the processes included in conversion process 22 to the image quality of the destination camera have conversion contents corresponding to the destination camera.

[0055] RGB-NR B-8 and YC-NR B-11 can be omitted.

[0056] The image that has been subjected to conversion processing 22 to convert to the image quality of the destination camera is output from the image quality converter 20. The image that is output from the image quality converter 20 is called a converted image. The image signal of the converted image is a nonlinear signal in the YCbCr format.

[0057] In this embodiment, the processing in the image quality converter 20 includes a conversion process 21 to a linear signal. Then, the linear signal that has been converted into a linear signal through the conversion process 21 is subjected to a conversion process 22 to the image quality of the conversion destination camera. Therefore, the image quality of the converted image can be converted into an image quality that is close to the image quality of the learning image.

[0058] As described above, in the surgical image processing device 1, the output image of the camera 102 is subjected to image quality conversion processing in the image quality converter 20. The output image is converted into the image quality of the surgical image learned by the inference model by the image quality conversion processing in the image quality converter 20. The image quality conversion processing in the image quality converter 20 includes conversion processing to a linear signal, offset correction, gain correction, white balance correction, and conversion processing to the image quality of the destination camera.

[0059] The above-described means of signal processing and the order of signal processing performed by image quality converter 20 are merely examples. The means of signal processing and the order of signal processing within camera 102, which is the source camera, differ depending on camera 102. Therefore, it is preferable to change the type, means, order, etc. of signal processing required for linear signal conversion depending on camera 102.

[0060] Similarly, with regard to the conversion process to convert image quality into that of the conversion destination camera, it is preferable that the type, means, and order of signal processing be changed according to the conversion destination camera.

[0061] In the above description, the image quality conversion process is performed on RGB signals (RGB format) and YC signals (YCbCr format). The image quality conversion process may be performed on all RGB signals. The image quality conversion process may also be performed on color space signals other than RGB signals and YC signals.

[0062] The image quality conversion process may be realized by software or by hardware such as a signal processing LSI.

[0063] (Method of estimating inverse gamma) Gamma -1A method of estimating an inverse gamma in the inverse gamma C-2' will be described below. Fig. 5 is a diagram showing a conventional method of estimating an inverse gamma. Fig. 6 is a diagram showing a method of estimating an inverse gamma in this embodiment.

[0064] In order to estimate a gamma curve from the endoscopically captured image 120, the signal level of the endoscopically captured image 120 and the absolute value of the sensor output are required. The absolute value of the sensor output may be a ratio to saturation.

[0065] When the absolute value of the sensor output is unknown, a graph 130 is created in which the reflectance of each patch 125 of the gradation chart 124 is used as an input and the signal level of the endoscopic image 120 is used as an output.

[0066] The density of each patch 125 of the gradation chart 124 is predefined. The density may be a reflectance. The gradation chart 124 may be a reflective type or a transmissive type. The light source is a light source whose brightness can be changed, or a luminance box.

[0067] The endoscopically captured image 120 is captured at a reference brightness. The signal level of the endoscopically captured image 120 is the average value of the green color within the frame 122 of each patch 121 in the endoscopically captured image 120.

[0068] Graph 130 is a diagram showing a gamma curve 131. The X-axis of graph 130 is the reflectance of each patch 125 of the gradation Chart 124 as an input. The Y-axis of graph 130 is the signal level of each gradation in each patch 121 of the endoscopically captured image 120 as an output.

[0069] In the graph 130, values ​​are plotted for 24 patches 121 contained in one endoscopically captured image 120.

[0070] As shown in graph 130, a part of a gamma curve can be generated by a conventional inverse gamma estimation method. However, the reflectance used in the input is not the absolute value (relative to saturation) of the sensor output. Therefore, the plot shown in graph 130 is not a curve that matches the input and output. Therefore, the gamma curve cannot be estimated from the plot shown in graph 130.

[0071] The inverse gamma estimation method of this embodiment will be described with reference to Fig. 6. In the following description, differences from the conventional inverse gamma estimation method shown in Fig. 5 will be described.

[0072] In the conventional inverse gamma estimation method, the endoscopic captured image 120 is captured with a reference brightness. In contrast, in the inverse gamma estimation method of the present embodiment, the endoscopic captured image 120 is captured with a brightness sweep. The sweep range is, for example, from ×0.25 to ×2.

[0073] In addition, in the conventional inverse gamma estimation method, the input is the reflectance of each patch 125 of the gradation Chart 124. In contrast, in the inverse gamma estimation method of this embodiment, the input is a value obtained by multiplying the reflectance of each patch 125 of the gradation Chart 124 by the sweep ratio of brightness in endoscopic imaging. The intersection 127 in Fig. 6 means multiplying the reflectance of each patch 125 of the gradation Chart 124 by the sweep ratio of brightness in endoscopic imaging.

[0074] A graph 133 in FIG. 6 is a graph showing a gamma curve 134 obtained by the inverse gamma estimation method of this embodiment.

[0075] In the inverse gamma estimation method of this embodiment, an endoscopic image is captured by sweeping the brightness. The input is the reflectance x brightness sweep ratio. In addition, the graph 130 plots the values ​​of 24 patches 121 for each number of images in which the brightness has been swept. As described above, in the inverse gamma estimation method of this embodiment, the signal levels of all patches of all endoscopic captured images are plotted, and the point at which the captured image (output side) becomes saturated is accurately determined.

[0076] The point at which the captured image (output side) saturates is indicated by point 136. The point at which the output saturates is set as the input saturation point 136. The gamma curve 134 is created based on this saturation point 136. A frame 137 is drawn based on the saturation point 136 and an output value 138 when the input is the value of the saturation point 136. The gamma curve 134 is a graph normalized with the Max of the X-axis of the frame 137 set to 1.0. By defining point 136 as the input x 1 (=saturation point), it is possible to determine the absolute value of the input side (sensor saturation ratio). This makes it possible to estimate the gamma curve 134.

[0077] In addition, by capturing an endoscopic image with a swept brightness and plotting many coordinates, intermediate chart gradations can be interpolated, thereby improving the estimation accuracy of the gamma curve 134.

[0078] (Image quality converter) In the image quality converter 20 described with reference to Fig. 4, more accurate image quality conversion can be performed by appropriately adjusting the following items. Note that the following is an example, and other adjustment items may be provided. ·Menu item Gain settings: 0dB~MaxdB White balance: G / R, G / B Contrast: High / Normal / Low Color tone adjustment: R emphasis, B emphasis step (e.g. 5 out of 10) Saturation adjustment: Strong / Standard / Weak etc. Edge emphasis: Strong / Standard / Weak etc. Noise reduction: Strong / Standard / Weak etc.

[0079] Regarding the signal processing flow, the following items can be exemplified. Signal processing flow Gamma Curve: By Contrast CCM determinant: per gamma curve Post-gamma color correction formula: hue adjustment, saturation adjustment

[0080] The menu items shown are common to all camera models. You can check in advance how each menu item will change depending on the item, so knowing the current settings makes it easy to change the image quality.

[0081] Considering the degree of influence on inference, the signal processing flow is illustrated mainly with respect to items related to color reproduction. Regarding the signal processing flow, if setting information linked to gamma is disclosed, conversion will be easier.

[0082] With this information about the camera, a more appropriate image quality converter can be created.

[0083] (White Balance) White balance correction E-3 will now be described. In the image quality conversion equations for converting image quality from the source camera 102 to the destination camera, the conversion process to a linear signal includes a conversion equation that can be fixed by the camera and a conversion equation that needs to be variable depending on the shooting conditions of the source camera 102.

[0084] The latter conversion formulas that need to be variable include the conversion formula for white balance compensation, the conversion formula for exposure compensation that includes gain multiplication, and conversion formulas related to these conversion formulas. In addition, edge emphasis linked to AF is also included in the conversion formulas that need to be variable.

[0085] The white balance may be set manually by the user or may be left at the manufacturer's default setting. When the white balance is set manually, the white balance may not be appropriate. An example of a cause of an inappropriate white balance is the influence of external light.

[0086] The white balance set by the manufacturer at the time of shipment may not necessarily be the appropriate white balance. The white balance may not be appropriate due to individual differences between cameras, changes in parts over time, etc.

[0087] If the image quality is converted with the white balance not set properly, color shifts will occur not only in white subjects but also in subjects with low saturation, making it impossible to convert the image quality of the output image into an image quality that is close to the image quality of the surgical image that the inference model learned.

[0088] Unlike endoscopic cameras, general cameras use external light to perform white balancing. Endoscopic cameras capture images of the inside of the body using light emitted from the tip of the camera. Endoscopic cameras have their own light source. Therefore, white balancing must be performed using the light source of the endoscope.

[0089] In the method of correcting the white balance correction value in this embodiment, a white balance checking jig 150 is used. The white balance checking jig 150 will be described. Fig. 7 is a diagram showing the white balance checking jig 150 in this embodiment. A cross-sectional view 158 of the white balance checking jig 150 is also shown in Fig. 7.

[0090] 7, the white balance checking jig 150 includes a cylindrical jig body 151. The shape of the jig body 151 may be a rectangular parallelepiped, a cone, or the like.

[0091] An endoscope insertion port 156 is formed on one bottom surface 152 of the jig body 151. The endoscope insertion port 156 is a through hole for inserting the tip of the endoscopic camera 102 into the jig body 151. The endoscope insertion port 156 may be shielded from light by a light shielding ring or the like.

[0092] An achromatic color chart 160 (achromatic subject) is placed on the other bottom surface 154 inside the jig body 151. The achromatic color chart 160 is a member that is colored gray with a predetermined brightness.

[0093] The white balance checking jig 150 can be separated into a number of parts. Fig. 8 is a diagram showing the white balance checking jig 150 in a separated state. The jig body 151 is separated into a tip chart part 168, a housing part 166, and an endoscope insertion port part 164.

[0094] The leading edge chart portion 168 including the achromatic chart 160 can be removed from the housing portion 166 in the same way as removing a cap. This makes it easy to clean the inside of the jig body 151.

[0095] The endoscope insertion port 164 including the endoscope insertion port 156 can also be removed from the housing 166. Therefore, it can be replaced with an endoscope insertion port 164 having a different endoscope insertion port 156 formed therein depending on the manufacturer, model, tip diameter, shape, and the like of the camera 102 (endoscope camera).

[0096] Taking into consideration autoclave resistance and light-shielding properties, the exterior parts of each part can be made of metal materials such as aluminum, CFRP (carbon fiber reinforced plastic), and light-shielding polypropylene, etc. The achromatic color chart 160 can be made by coloring the inside of a polypropylene sheet gray.

[0097] The method of correcting the white balance correction value in this embodiment uses a white balance checking jig 150 that is attached to the tip of the camera 102 (endoscopic camera), can completely block light, and has a certain or larger area in the center of the angle of view composed of achromatic color (gray). The tip of the camera 102 is inserted into the inside of the jig body 151 through the endoscope insertion port 156. The white balance is checked using an achromatic chart 160. Specifically, a white balance error is measured, and the white balance correction value of the image quality converter 20 is corrected based on the error. The image quality converter 20 may include a white balance error generating means that generates a white balance error.

[0098] The white balance check is started in response to an instruction from the user or when the camera 102 automatically determines from an image that the white balance check tool 150 is attached.

[0099] (Low saturation pixels, low saturation areas) As another embodiment, there is a method that does not use the white balance checking jig 150. In this method, the white balance error is estimated based on the R / G and B / G signal ratios of a chromatic subject with low saturation in the video of the surgical field. The white balance correction value of the image quality converter 20 is corrected based on the estimated error.

[0100] Specifically, it judges low saturation pixels, which are pixels of chromatic colors with low saturation in the output image 110, or low saturation areas, which are areas of chromatic colors with low saturation in the output image 110. The image quality converter 20 may be provided with low saturation judgment means for making this judgment.

[0101] Next, a white balance gain for the low saturation pixels or a white balance gain for the low saturation region is generated. The white balance gain generating means for generating this white balance gain may be provided in the image quality converter 20.

[0102] Then, a white balance error for a low saturation pixel or a white balance error for a region is generated according to the color temperature. The white balance error generating means for generating the white balance error may be provided in the image quality converter 20.

[0103] The image quality converter 20 corrects the white balance correction value with the generated white balance error.

[0104] As described above, instead of detecting specific parts of the surgical field, the image is converted to a color space or equivalent color gamut, and white balancing is performed by focusing on low saturation areas in that space. The surgical field image can also be used to perform color correction processing.

[0105] (Skin color confirmation chart) As another embodiment, there is a method of correcting the white balance correction value using a white balance checking tool 150 equipped with a skin color checking chart 162 instead of the achromatic color chart 160. The parts that differ from the embodiment shown in FIG. 8 will be described. FIG. 9 is a diagram showing a white balance checking tool 150 of another embodiment. As shown in FIG. 9, a skin color checking chart 162 is arranged on a bottom surface 154 inside a jig body 151. The skin color checking chart 162 is a chart on which skin colors of different lightness, hue angles, and saturation in multiple tones are arranged.

[0106] When using a white balance check tool 150 (skin color reproduction check tool) equipped with a skin color check chart 162, an error in skin color reproduction is measured using the skin color check chart 162. A skin color reproduction error generating means for measuring a skin color reproduction error may be provided in the image quality converter 20. Based on the measured error, the white balance correction value of the image quality converter 20 and the color conversion process are corrected.

[0107] In each of the above-described embodiments, the operator may be notified of deviations in the white balance setting, deviations due to aging, etc. based on the obtained information, thereby supporting the use of the endoscopic camera in an appropriate state.

[0108] (Exposure compensation) The exposure compensation E-2 will be explained. In the conversion formula of the exposure compensation, basically, the ratio of the exposure convergence level of each camera of the conversion source and the conversion destination is used, and the exposure compensation ratio that is fixed for the combination of the conversion source and the conversion destination camera is used. The exposure convergence level is the brightness level at which the signal in the linear region becomes the target.

[0109] However, for example, in the case of a camera equipped with an automatic exposure or automatic brightness adjustment, the brightness of the image varies from camera to camera depending on the conditions of the subject. For example, in a scene with a large amount of light and dark, the brightness varies from camera to camera depending on the size of the main subject and the magnitude of the difference between light and dark.

[0110] Therefore, when converting the image quality of the output image from the source camera, if exposure correction is performed on the image converted into a linear signal using only a fixed exposure correction ratio that corresponds to the source and destination cameras, the image may be too bright or too dark depending on the gamma characteristics of the destination camera and the subject conditions, and it will not be possible to convert the image into an image quality that approximates the image quality of the surgical image learned by the inference model.

[0111] An example of exposure correction will be described with reference to FIG. 10. FIG. 10 is a diagram showing an example of conventional exposure correction. FIG. 10 shows an example in which an image becomes too bright after image quality conversion. Image 200 shows the image before conversion. Image 202 shows the image after conversion. Image 200 has a large difference in brightness within the image. Image 200 is an example in which dark areas 204 are raised and bright areas 206 are reproduced to the best of their ability. First, the image signal of this image 200 is converted into a linear signal. The conversion is performed by performing a process that is the inverse gamma of the gamma of the source signal. Exposure correction E-2 is performed on the obtained linear signal. The exposure correction ratio is the exposure convergence level of the destination signal / the exposure convergence level of the source signal. The exposure correction ratio will be described later. After exposure correction E-2, gamma processing of the destination signal is performed to obtain a converted image 202.

[0112] In the example shown in Fig. 10, the gamma of the destination image is raised significantly, so the signal level is lowered by exposure compensation. However, even if the exposure level is lowered by exposure compensation, the source image has a large difference in brightness, and the bright area 206 is bright, so the gradation of the bright area 206 cannot be reproduced with a fixed exposure compensation ratio. In endoscopic surgery, the light is directed at the area to be viewed, so the bright side is more problematic than the dark side.

[0113] Before describing the exposure correction of this embodiment, a method for determining a fixed exposure correction ratio will be described with reference to Fig. 11. Fig. 11 is a diagram showing a method for determining a fixed exposure correction ratio. In the example shown in Fig. 11, the brightness of the output image is set to be the same between the source camera and the destination camera. The brightness of the output image is 140 levels out of 256 levels.

[0114] First, a gray chart 210 with a reflectance of 18% is photographed with the source camera and the destination camera. The source camera and the destination camera have different exposure convergence levels. An image 212 taken with the source camera has a bright exposure convergence level. On the other hand, an image 216 taken with the destination camera has a dark exposure convergence level. Therefore, exposure correction is performed by setting the exposure correction ratio to the exposure convergence level of the destination / the exposure convergence level of the source. Then, gamma processing is performed for each. This makes the linear signal at the time of sensor output a nonlinear signal. This exposure correction makes the output images 214 and 218 from the cameras have the same brightness.

[0115] Exposure correction according to this embodiment will be described below. Fig. 12 is a diagram showing an example of exposure correction. In the following description, differences from the matters described with reference to Fig. 10 will be described.

[0116] For the conversion formula of exposure compensation in the image quality converter 20, the maximum and minimum signal levels at which the converted signal is at an appropriate level in the gamma of the conversion destination are calculated in advance. An appropriate level is a level at which the bright areas are not too bright to be reproduced, and the dark areas are not too dark to be reproduced. The range from the minimum signal level to the maximum signal level at the appropriate level is called a predetermined level range. This predetermined level range can also be called an optimal level range. When a signal greater than (or equal to) this maximum signal level exceeds a predetermined allowable rate in the source image, or when a signal smaller than (or equal to) the minimum signal level exceeds a predetermined allowable rate, the exposure compensation ratio in the image quality converter 20 is corrected.

[0117] Specifically, as shown in the flow in Fig. 12, a first preset 220 predetermines the maximum and minimum values ​​of the input level at which the converted signal is appropriate (input level calculation means). Also, a second preset 222 predetermines the allowable rate of pixels above the maximum value / below the minimum value (allowable value setting means). In a histogram 224 of the signal of the image 200 before conversion, the frequency of pixels above the maximum value or below the minimum value determined in the first preset 220 is calculated (pixel ratio calculation means). The example shown in Fig. 12 shows a case where the frequency of pixels above the maximum value is calculated. Box 226 indicates the frequency of pixels that are too bright after conversion.

[0118] Then, in decision step 228, it is determined (comparison means) whether the frequency of pixels that are too bright after conversion, shown in box 226, exceeds the tolerance rate defined in the second preset 222. And if the determination is Yes, the correction ratio is reduced, i.e., the gain is reduced.

[0119] The same is true for the minimum value defined in the first preset 220. If the frequency of pixels that are too dark after conversion exceeds the tolerance rate defined in the second preset 222, the correction ratio is increased, i.e. the gain is increased.

[0120] The endoscopic camera shines light on the area to be viewed, i.e., the area where surgery is performed. Therefore, it is more important to suppress blown-out highlights and color shifts on the brighter side than on the darker side. If AE (auto exposure) is implemented in the image quality converter 20, it will require a lot of work, and in conditions where there is no difference in brightness, the AE on the camera side will be sufficient, so installing AE in the image quality converter 20 would be overkill.

[0121] As described above, exposure compensation is performed at a fixed ratio according to the camera's AE, and if the image becomes too bright due to the gamma of the destination camera, the fixed ratio can be corrected to obtain optimal exposure with little cost and resources.

[0122] (Selection of conversion formula) The conversion formula for image quality conversion, i.e., the selection of the image quality conversion formula, will be explained. Image quality conversion formulas differ depending on the camera used. Therefore, the conversion formula is set by the user from a database that stores a series of conversion formulas for conversion to the image quality (spatial frequency, brightness, color tone, etc.) equivalent to the conversion destination, or the conversion formula is automatically set by the inference model selected by the user. If the user sets the conversion formula incorrectly, the appropriate conversion formula will not be selected. As a result, it will not be possible to convert to image quality that is close to the image quality of the surgical image that the inference model has learned.

[0123] In the selection of the image quality conversion formula of this embodiment, the characteristics (color distribution, histogram, spatial frequency, etc.) of the image output from the camera are analyzed, and the characteristics are stored in advance in a database that stores conversion formulas. Then, when selecting an image quality conversion formula, it is determined which camera in the database matches the characteristics of the image output from the camera, and a conversion formula registered in the database is automatically selected. Alternatively, if the conversion formula selected by this means differs from the conversion formula set by the user or automatically set by the inference model selected by the user, a warning message is displayed to prompt the user to confirm whether the user's settings are appropriate. Image quality characteristics can be registered not only for each camera, but also for each menu setting for each camera.

[0124] The following description will be given with reference to the drawings. Fig. 13 is a diagram showing the selection of a conventional image quality conversion formula. Fig. 14 is a diagram showing the selection of an image quality conversion formula of this embodiment. In Figs. 13 and 14, an image 230 indicates an image to be converted, and an image 232 indicates an image to be converted.

[0125] As shown in Fig. 13, conventionally, the conversion formula applied to the image quality converter 20 is selected by the user from among the conversion formulas stored in the image quality conversion formula DB 234, or is automatically selected from an inference model. In the example shown in Fig. 13, the source camera is camera A, the destination camera is camera B, and the conversion formula AB-1 is selected. Here, it is possible that the selection of the conversion formula AB-1 is incorrect.

[0126] In contrast, in the selection of an image quality conversion formula in this embodiment, as shown in FIG. 14, a conversion formula is automatically selected by referring to image quality characteristic information stored in advance in the image quality conversion formula DB 234. Examples of image quality characteristic information are histogram and color space mapping. When selecting an image quality conversion formula, image quality characteristics are generated based on information from the image quality converter 20 and the like (analysis means). It is preferable that the example of the image quality characteristic is the same as the image quality characteristic information. Examples of the image quality characteristic are histogram and color space mapping. Based on the image quality characteristics acquired from the image quality converter 20, the image quality characteristic information stored in the image quality conversion formula DB is referred to (determination means), and a conversion formula is automatically selected. This enables the selection of a conversion formula with fewer errors.

[0127] We will explain the selection of image quality conversion formula in more detail. The image quality of a camera varies depending on the image sensor, signal processing LSI, settings of the image sensor and signal processing LSI, exposure convergence level of automatic exposure, and white balance, and each company produces different images. Also, even with the same endoscope system, different image quality is generated depending on the menu settings.

[0128] On the other hand, the image quality itself can be evaluated by analyzing the image using indices such as color reproduction, brightness, contrast, tone reproduction, resolution, edge strength, and noise, and different evaluation values ​​can be obtained for each image quality.

[0129] For example, for color reproduction, it is possible to estimate the camera type and menu setting candidates by plotting the image in a color space such as L*a*b* and quantifying the center position and distribution tendency, etc. In the case of surgery, the organs that will be in the surgical field are determined by the surgical area and surgical procedure, so the judgment index may be changed depending on the surgical target. In addition, the state of the color space of the surgical field may change depending on the condition of the patient in the target case (e.g., a lot of fat) and the situation during surgery (e.g., heavy bleeding), so the judgment index and image quality conversion method may be changed accordingly.

[0130] In addition, for example, by estimating candidate camera and menu settings from numerical values ​​of contrast and tone reproduction characteristics obtained from the image histogram and judging them together with the estimated results based on color reproduction, it is possible to determine the camera type and menu settings with greater accuracy.

[0131] During surgery, the camera may be removed from the body to clean the lens. This may result in an image that differs from the actual surgical field being output. Therefore, the image quality characteristics may be judged only when the image matches the color conditions specific to the surgical field (organs, blood vessels, etc.). In addition, the judgment results may be stabilized over time by, for example, applying a low-pass filter to the judgment results.

[0132] The technical scope of the present disclosure is not limited to the above-described embodiment. The above-described embodiment may be modified in various ways and may be improved in various ways. Furthermore, the modified or improved forms are included in the technical scope of the present disclosure.

[0133] For example, the image quality converter 20 may have a means for performing image quality conversion including at least one of color conversion, brightness conversion, noise reduction, and edge enhancement on the nonlinear signal before converting the output image 110 into a linear signal. This enables image quality conversion with a nonlinear signal in accordance with the configuration of the source camera.

[0134] Furthermore, conversion from a nonlinear signal to a linear signal and conversion from a linear signal to a nonlinear signal can be omitted as appropriate depending on the configuration of the camera, etc. This may improve conversion accuracy or inference accuracy.

[0135] In the above embodiment, each process is described by taking as an example a scene in which the image quality of the output image 110 is converted to an image quality that is close to the image quality of the surgical image learned by the inference model. Each process may be used to generate a training image for the inference model to learn.

[0136] The process flow when image quality conversion is used to generate learning images is, for example, as shown below. Although the following uses a still image as an example, the image may be a video. "Acquire surgical still images -> Convert the image quality of the acquired surgical still images -> Add annotations (e.g., correct answer data) to the surgical still images after image quality conversion -> Use this data as training data to create an inference model through machine learning." Examples of annotations include various items such as evaluation of the level of surgery, information about the body, and information about surgical instruments.

[0137] During learning, images or videos in which at least one of the source image quality conversion and the destination image quality conversion is different may be mixed for learning.

[0138] 1B, the image quality converter 20 may be provided separately from the surgery support system 100. The image quality converter 20 may also be provided with an instruction unit 31. The image quality converter 20 is connected to a learning unit 32.

[0139] In the above embodiment, the surgical image processing device has been described as an object. The surgical image processing device may be a program that causes the surgical image processing device to function as each unit.

[0140] (Conversion to image quality of other endoscopes) The conversion of image quality in the image quality converter 20 described with reference to Fig. 4 etc. will be specifically described. First, the conventional technology will be described. Fig. 15 is a diagram showing an example of conventional image quality conversion. Fig. 15 shows an example of converting images of various image qualities of an endoscope A into images of various image qualities of an endoscope B.

[0141] Even images captured by the same endoscope may have different image quality. In the example shown in Fig. 15, endoscope A has three types of image quality: image quality A-1, image quality A-2, and image quality A-3. Image quality A-1, image quality A-2, and image quality A-3 have different white balance and image quality settings set by the user. Among image quality A-1, image quality A-2, and image quality A-3, image quality A-1 is set as the default image quality.

[0142] The same is true for endoscope B. Endoscope B has image quality B-1 and image quality B-2. Image quality B-1 and image quality B-2 have different white balances. Image quality B-1 is set as the default image quality.

[0143] When converting image quality from endoscope A to endoscope B, there are six possible routes for image quality conversion, from route 1 (R1) to route 6 (R6), as shown in Fig. 15. Route 1 (R1) is a route for converting image quality A-1 to image quality B-1, route 2 (R2) is a route for converting image quality A-1 to image quality B-2, route 3 (R3) is a route for converting image quality A-2 to image quality B-1, route 4 (R4) is a route for converting image quality A2 to image quality B-2, route 5 (R5) is a route for converting image quality A-3 to image quality B-1, and route 6 (R6) is a route for converting image quality A-3 to image quality B-2.

[0144] 16 is a diagram showing steps of a conventional conversion from image quality A-3 to image quality B-2. The conversion shown in FIG. 16 corresponds to the conversion of root 6 (R6) in FIG.

[0145] As shown in FIG. 16, route 6 (R6) includes four steps: inverse gamma H-1, white balance adjustment H-2, color conversion H-3, and gamma H-4. The processing content of each step is the same as that described with reference to FIG. 4, etc. When converting image quality from image quality A-3 to image quality B-2, first, in the inverse gamma H-1 step, the image is converted from non-linear to linear. Next, white balance adjustment H-2 and color conversion H-3 are performed. Furthermore, in the gamma H-4 step, the image is converted from linear to non-linear. As a result, image quality A-3 is converted to image quality B-2.

[0146] Of the four steps included in route 6 (R6), inverse gamma H-1, white balance adjustment H-2, color conversion H-3, and gamma H-4, inverse gamma H-1 is common to image quality A, white balance adjustment H-2 and color conversion H-3 are dedicated to conversion from image quality A-1 to image quality B-2, and gamma H-4 is common to image quality B. "Common to image quality A" means that it can be used in common for various image qualities such as image quality A-1, image quality A-2, and image quality A-3 in endoscope A. Similarly, "common to image quality B" means that it can be used in common for various image qualities such as image quality B-1 and image quality B-2 in endoscope B.

[0147] In contrast, the white balance adjustment H-2 and color conversion H-3 are dedicated to individual conversions, because color conversion depends on white balance and therefore different processing is required for each white balance.

[0148] Like white balance adjustment H-2 and color conversion H-3, each conversion route includes a step that performs a different conversion for each combination of conversions. Therefore, a conversion formula for converting image quality is required for each route. In the example shown in Figure 15, six conversion formulas are required, one for each of the six routes from route 1 (R1) to route 6 (R6).

[0149] In this way, when an image quality converter is used to support many endoscopes and endoscope systems, if conversion formulas are created individually for each endoscope, the number of conversion formulas becomes enormous.

[0150] The image quality converter 20 of this embodiment will be described with reference to Fig. 17 and Fig. 18. Fig. 17 is a diagram showing an example of image quality conversion of this embodiment. In the following description, differences from the conventional image quality converter described with reference to Fig. 15 and Fig. 16 will be mainly described. Items that are not particularly described in the following description can be made the same as those of the conventional image quality converter.

[0151] As shown in Fig. 17, in the image quality converter 20 of this embodiment, image quality conversion is performed via the respective default image quality of endoscope A and endoscope B. Image quality other than the default in the endoscope of the source is first converted to the default image quality. Then, the default image quality in the endoscope of the source is converted to the default image quality in the endoscope of the destination. After that, as necessary, it is converted to the image quality other than the default in the endoscope of the destination.

[0152] In the example shown in Figure 17, image quality A-2 is converted to image quality A-1 by route 11 (R11), and image quality A-3 is converted to image quality A-1 by route 12 (R12). Image quality A-1 is converted to image quality B-1 by route 1 (R1).

[0153] When it is necessary to convert the image quality to B-2, the image quality B-1 is converted to the image quality B-2 by route 21 (R21).

[0154] In the image quality converter 20 of this embodiment, the image quality is not directly converted from each source image quality to each destination image quality, but is converted via the default image quality of each endoscope A and endoscope B. This makes it possible to reduce the number of types of conversion formulas to be prepared.

[0155] In the conventional example shown in Fig. 15, there are six types of conversion routes, and therefore six types of conversion formulas need to be prepared. In contrast, in the example of this embodiment shown in Fig. 17, there are only four types of conversion routes, and therefore four types of conversion formulas need to be prepared. In addition, as will be described later, the conversion formula can be common to root 11 (R11) and root 12 (R12). Therefore, in the example of this embodiment, the types of conversion formulas that need to be prepared can be three.

[0156] In addition, image quality conversion between different endoscopes places a greater burden on the conversion process than image quality conversion within the same endoscope. With the image quality converter 20 of this embodiment, image quality conversion between different endoscopes can be limited to conversion via route 1 (R1). This further reduces the burden of preparing a conversion formula.

[0157] An example of image quality conversion in the image quality converter 20 of this embodiment will be described with reference to Fig. 18. Fig. 18 is a diagram showing steps of conversion from image quality A-3 to image quality B-2 of this embodiment. The conversion shown in Fig. 18 corresponds to the conversion of root 12 (R12), root 1 (R1) and root 21 (R21) in Fig. 17.

[0158] As shown in Figure 18, route 12 (R12) includes two steps: inverse gamma H-21 and white balance adjustment H-22. Route 1 (R1) includes color conversion H-23. Route 21 (R21) includes two steps: white balance adjustment H-24 and gamma H-25.

[0159] Image quality A-3 is converted to image quality A-1 through the steps included in route 12 (R12). Image quality A-1 is converted to image quality B-1 through color conversion H-23 included in route 1 (R1). Image quality B-1 is converted to image quality B-2 through the steps included in route 21 (R21).

[0160] Of the three routes involved in converting from image quality A-3 to image quality B-2, route 12 (R12), route 1 (R1), and route 21 (R21), route 12 (R12) is common to image quality A, route 1 (R1) is exclusive to converting from image quality A-1 to image quality B-2, and route 21 (R21) is common to image quality B.

[0161] As shown by the fact that root 12 (R12) is common to image quality A and root 21 (R21) is common to image quality B, conversion within the same endoscope can use the same conversion formula regardless of the image quality of the source and the image quality of the destination. In the example of endoscope A, the conversion formula of root 11 (R11) and the conversion formula of root 12 (R12) can be the same. This is because conversion processes other than color conversion are not dependent on the image quality of the source and the image quality of the destination.

[0162] In the image quality converter 20 of this embodiment, by preparing a conversion formula within the endoscope at the source endoscope, a conversion formula from the default image quality at the source endoscope to the default image quality at the destination endoscope, and a conversion formula within the endoscope at the destination endoscope, it is possible to convert the image quality from the source endoscope to the image quality of the destination endoscope.

[0163] (summary) Of the conversion processes performed by the image quality converter 20, color conversion requires conversion according to the image quality of the source and the image quality of the destination. Therefore, different conversions are required depending on the combination of the endoscope and its image quality. For example, different white balances require different color conversions. In other words, a conversion formula must be prepared for each combination of the source endoscope and the destination endoscope.

[0164] On the other hand, in conversion processes other than color conversion including white balance, the conversion formula does not have dependency on the endoscope of the conversion source and the endoscope of the conversion destination. Therefore, independent conversion is possible for each of the conversion source and the conversion destination.

[0165] Therefore, in the source endoscope, in either case of image quality, first, the image quality is converted to the default setting of the source endoscope, then converted to the default setting of the destination endoscope, particularly color conversion, and then converted from the default setting of the image quality of the destination endoscope to the desired image quality.

[0166] As a result, in the image quality converter 20 of this embodiment, only one conversion formula between endoscopes is required.

[0167] On the other hand, the white balance conversion is present in both the source endoscope and the destination endoscope. However, the conversion processes other than the color conversion are simpler than the conversion process in the color conversion.

[0168] Furthermore, conversion processes other than color conversion are common processes within each endoscope and are not dependent on image quality, which makes it possible to prevent the number of conversion formulas from becoming too large.

[0169] (If there are three or more endoscopes) Conventional image quality conversion in the case where there are three or more endoscopes will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example of conventional image quality conversion in the case where there are three endoscopes. When converting the image quality of a source endoscope to the image quality of a destination endoscope, if the number of endoscopes to be supported increases, a new conversion formula must be created on the endoscope that was previously supported. This is the same whether conversion is performed individually for each image quality as in the conventional case, or conversion is performed via the default image quality of each of the source endoscope and destination endoscope.

[0170] Fig. 19 shows an example of conversion when endoscope C is added to the conversion between endoscope A and endoscope B shown in Fig. 15 etc. As shown in Fig. 19, the addition of endoscope C makes it necessary to prepare new conversion formulas corresponding to each of the routes, route 31 (R31) from endoscope A to endoscope C and route 32 (R32) from endoscope B to endoscope C. In this way, the more endoscope models there are, the greater the number of conversion combinations becomes, and the corresponding labor required to prepare new conversion formulas becomes enormous.

[0171] (If there are three or more endoscopes) Image quality conversion according to this embodiment when there are three or more endoscopes will be described with reference to Fig. 20. Fig. 19 is a diagram showing an example of image quality conversion according to this embodiment when there are three endoscopes. As described with reference to Fig. 19, if image quality conversion is performed on the premise of direct conversion from the conversion source to the conversion destination, it is necessary to add conversion formulas for all endoscopes every time the number of models increases.

[0172] In the image quality converter 20 of this embodiment, conversion is first performed from the source endoscope to intermediate image quality or standard image quality. Then, conversion is performed from the intermediate image quality or standard image quality to the destination image quality. This means that the conversion formula for each endoscope only needs to correspond to the image quality of the endoscope and intermediate image quality, or the image quality of the endoscope and standard image quality. As a result, even if an endoscope is added, it is only necessary to add a conversion formula between the added endoscope and intermediate image quality, or a conversion formula between the added endoscope and standard image quality.

[0173] In the example shown in Fig. 20, it is assumed that an endoscope C is added to endoscopes A and B. The conversion formulas corresponding to route 41 (R41) and route 42 (R42) are prepared before endoscope C is added. In the example shown in Fig. 20, when endoscope C is added, it is only necessary to newly prepare a conversion formula corresponding to route 43 (R43). In other words, it is possible to eliminate the need for a conversion formula on the endoscope A side and the endoscope B side.

[0174] (Medium and standard quality) The medium image quality and the standard image quality will now be described. The intermediate image quality refers to the image quality of any representative endoscope, or the average or median of multiple representative endoscopes, or image quality generated according to a ratio arbitrarily set for each endoscope. For example, by setting the average value of multiple representative endoscopes as the intermediate image quality, it is possible to reduce the amount of correction from the intermediate image quality to each endoscope.

[0175] Standard image quality is defined as the image quality of the raw data from an endoscope's image sensor, or an image quality close to that. For example, the image quality obtained from the output of a 4K RGB sensor or a 4K RGBW sensor (a sensor capable of simultaneous exposure to visible and infrared light) is considered standard image quality. Unnecessary signal processing is not applied to raw data. Therefore, linear performance is guaranteed for raw data. As a result, image quality can be easily converted between endoscope models.

[0176] The intermediate image quality is also restored to a linear signal in the same way as the standard image quality. However, the restoration may not be perfect. Therefore, a certain amount of conversion error may be introduced into the intermediate image quality. As a result, when using the intermediate image quality, it may be necessary to make adjustments while checking the conversion result.

[0177] (View image) The images displayed by a surgical image processing device with an image quality converter are either images with the original image quality or images after conversion to the destination image quality. In either case, when changing endoscopes, it is necessary to take into consideration the differences in the model during observation. This is because the tendency for color tone and contrast varies depending on the endoscope.

[0178] In addition, when displaying the results of AI analysis, the displayed colors may be difficult to see depending on the model of endoscope and the settings of white balance, color adjustment, etc. Although a method for selecting colors may be provided, even in such cases, if different colors are selected for each endoscope, it may become difficult to know which structure the displayed color refers to.

[0179] If the image quality converter has a function for converting image quality to the default image quality of the endoscope, it converts the image quality to the default image quality of the specified endoscope, and displays an image generated from the converted image. This allows observation to always be performed using images with image quality that is familiar to the user. In addition, the display color of the AI ​​analysis results can always be the same and can be selected as an easy-to-understand color. The display color of the AI ​​analysis results can also be automatically set according to the set intermediate image quality or standard image quality. It may be set to an easy-to-see display color, or it may be specified not to allow the selection of a hard-to-see display color.

[0180] If the image quality converter has the function of converting image quality to intermediate image quality, an image generated from the intermediate image quality image is displayed. Generally, intermediate image quality is based on neutral image quality with little emphasis when considering conversion to various models. This allows stable observation in any scene. When displaying intermediate image quality, a corrected image may be displayed according to the user's preferences. This allows the same color to be selected for displaying the AI ​​analysis results, making the display easier to understand. The above content also applies when using standard image quality instead of intermediate image quality.

[0181] To make the processing in the image quality converter visible, an intermediate image quality image may be displayed, for example, on a monitor. For example, the intermediate image quality is displayed as an image converted to a preferred or familiar image quality. The inferred drawing color may be difficult to see depending on the image quality of the endoscope. Therefore, by displaying the intermediate image quality image, the selected inferred drawing color can be made easy to see regardless of the endoscope. [Explanation of symbols]

[0182] 1. Surgical image processing device 10 Pretreatment section 12 Reasoning part 14 Drawing section 16 Display section 20 Image Quality Converter 31 Instruction section 32 Learning Department 100 Surgical Support System 102 Camera (source) (endoscopic camera) 104 Display device 110 Output Image 112 Converted Images 120 Endoscopic Images 121 Patch 122 Box 124 Tone Chart 125 Patches 127 Intersection 130 Graphs 131 Gamma Curve 150 White balance check tool 151 Jig body 152 One side of the bottom 154 The other bottom 156 Endoscope insertion port 158 Cross Section 160 Achromatic Color Chart 162 Skin Color Check Chart 164 Endoscope insertion port 166 Housing 168 Tip Chart Section 200 Images before conversion 202 Converted image 204 Dark part 206 Akabe

Claims

1. A surgical image processing device included in a surgical support system, a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by an arbitrary camera separate from the surgical image processing device; an inference unit including an inference model for analyzing a converted image when the image after image quality conversion is a converted image; a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image; Equipped with The image quality converter includes: a means for performing image quality conversion including at least one of color conversion and luminance conversion on a nonlinear signal before the output image is converted into a linear signal; Then, an inverse gamma processing means for converting the output image into a linear signal; means for performing image quality conversion on the linear signal, the image quality conversion including at least one of color conversion, luminance conversion, noise reduction, and edge enhancement; a gamma processing means capable of performing gamma processing on the converted image, the gamma processing means being capable of performing or not performing the gamma processing; Surgical imaging equipment.

2. The gamma processing means does not perform the gamma processing. The surgical image processing apparatus according to claim 1 .

3. The image quality converter includes: The method further includes a means for performing at least one of an offset adjustment, a brightness adjustment, and a white balance adjustment on the linear signal. The surgical image processing device according to claim 1 .

4. the output image includes a non-linear signal that has been gamma processed and a linear signal that has not been gamma processed; The image quality converter converts the image quality of the linear signal. The surgical image processing apparatus according to claim 1 .

5. the output image contains only linear signals that have not been gamma-processed; The image quality converter converts the image quality of the linear signal. The surgical image processing apparatus according to claim 1 .

6. The inference model is trained on an image obtained by subjecting the converted image to the gamma processing. The surgical image processing apparatus according to claim 1 .

7. In the inverse gamma processing means, A value obtained by multiplying a reflectance or transmittance at a plurality of points in a gradation chart in which a plurality of densities are defined by different brightness ratios when the surgical image is captured is input; In the graph, the signal level of the gradation at a plurality of points in the plurality of surgical images taken at different brightness ratios is output. The conditions for the inverse gamma processing are obtained from a graph normalized by setting the input at the point where the output is saturated to 1. The surgical image processing device according to claim 1 .

8. A surgical image processing device included in a surgical support system, a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by the camera; an inference unit including an inference model for analyzing a converted image when the image after image quality conversion is a converted image; a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image; Equipped with the camera is an endoscopic camera; The image quality converter includes: a white balance checking tool that uses a light source of the endoscope camera; a white balance error generating means for automatically determining whether the white balance checking tool is attached based on an image captured by the endoscope camera and generating a white balance error; and a means for correcting a white balance correction value using the generated white balance error. Surgical imaging equipment.

9. The inference model is trained on an image in which a white balance correction value has been corrected based on the white balance error. The surgical image processing apparatus according to claim 8 .

10. the camera is an endoscopic camera; The image quality converter includes: a low saturation determining means for determining a low saturation pixel, which is a pixel of a chromatic color with low saturation in the output image, or a low saturation region, which is a region of a chromatic color with low saturation in the output image; a white balance gain generating means for generating a white balance gain for a low saturation pixel or a white balance gain for a low saturation region; a white balance error generating means for generating a low saturation pixel white balance error or a region white balance error according to a color temperature; and a means for correcting a white balance correction value using the generated white balance error. The surgical image processing apparatus according to claim 8 .

11. The inference model is trained using an image in which a white balance correction value has been corrected based on the white balance error according to color temperature. The surgical imaging apparatus of claim 10.

12. the camera is an endoscopic camera; The image quality converter includes: a skin color reproduction confirmation tool attached to the tip of the endoscope camera, shielded from light, and having skin colors of different luminosity, hue angle, and saturation arranged in the center of the angle of view; a skin color reproduction error generating means for measuring a skin color reproduction error based on an instruction from a user or by automatically determining that the skin color reproduction confirmation tool is attached; and a means for correcting at least one of a white balance correction value and a color conversion process using the generated skin color reproduction error. The surgical image processing apparatus according to claim 8 .

13. The inference model is trained on an image in which at least one of a white balance correction value and a color conversion process has been corrected based on the skin color reproduction error. The surgical imaging system of claim 12.

14. A surgical image processing device included in a surgical support system, a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by the camera; an inference unit including an inference model for analyzing a converted image when the image after image quality conversion is a converted image; a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image; Equipped with The image quality converter includes: a means for calculating an input level such that the signal level of the output image after image quality conversion falls within a predetermined level range; means for setting an acceptance rate for pixels outside the predetermined level range; means for calculating a proportion of pixels of an input image that are outside the predetermined level range; means for comparing the ratio with the tolerance rate; and an exposure correction means for correcting an exposure correction ratio in accordance with the excess ratio when the ratio exceeds the allowable ratio. Surgical imaging equipment.

15. The inference model is trained on images with a corrected exposure compensation ratio. The surgical imaging system of claim 14.

16. A surgical image processing device included in a surgical support system, a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by the camera; an inference unit including an inference model for analyzing a converted image when the image after image quality conversion is a converted image; a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image; Equipped with The image quality converter includes: A database storing image quality characteristics for each of the cameras; means for analyzing characteristics of the output image from the camera; means for determining a match between the results of said analysis and the image quality characteristics stored in said database; The surgical image processing apparatus according to claim 1 , further comprising: a means for automatically selecting the conversion formula for image quality.

17. The inference model is trained on an image whose image quality has been transformed using the selected transformation formula. The surgical imaging system of claim 16.

18. The surgical image processing device included in the surgical support system includes: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by the camera; an inference unit including an inference model for analyzing a converted image when the image after image quality conversion is a converted image; a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image; By utilizing the functions of The image quality converter includes: calculating an input level such that the signal level of the output image after image quality conversion falls within a predetermined level range; Setting an acceptance rate for pixels outside the predetermined level range; calculating a percentage of pixels of the input image that are outside the predetermined level range; Comparing the percentage with the tolerance percentage; If the ratio exceeds the allowable ratio, the exposure compensation ratio is corrected according to the excess ratio; Exercising a function of converting the image quality of the output image of the surgical image acquired by the camera; program.

Images