Surgical image processing device

Abstract

The present invention provides a surgical image processing device 1 that converts the image quality of a photographed image into desired image quality. A surgical image processing device 1 comprises: a preprocessing unit 10 including an image quality converter 20 that converts the image quality of an output image of a surgical image captured by a camera 102; an inference unit 12 including an inference model for analyzing a converted image when an image with converted image quality is used as the converted image; and a drawing unit 14 that draws an image in which an analysis result by the inference model is reflected on the surgical image. The image quality converter 20 includes an inverse gamma processing means for converting the output image into a linear signal, a means for performing image quality conversion including at least one of color conversion, luminance conversion, noise reduction, and edge enhancement on the linear signal, and a gamma processing means for performing gamma processing on the converted image.

Description

Surgical image processing device

[0001] The present invention relates to a surgical image processing device.

[0002] 2. Description of the Related Art A technique is known for assisting surgery by acquiring surgical images of a surgery performed by a surgeon, analyzing the surgical images, and applying the analysis results to the surgical images.

[0003] Japanese Patent Laid-Open No. 2006-129999 describes that when processing an endoscopic image captured by an endoscopic device, the image quality is converted based on an image quality adjustment value set in the endoscopic device.

[0004] Patent No. 7130038

[0005] When converting the image quality of an endoscopic image captured by an endoscopic device to a predetermined image quality, the image quality adjustment value of the endoscopic device that captured the endoscopic image may not be clear. Furthermore, the gradation response characteristics of the image signal of the endoscopic image output from the endoscopic device are generally nonlinear. In such cases, no technology has been proposed for converting the image quality of the captured image to a desired image quality.

[0006] An object of the present invention is to provide a surgical image processing device that converts the image quality of a captured image into a desired image quality.

[0007] The surgical image processing device of the present invention is a surgical image processing device included in a surgical support system, and comprises: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera; an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is considered to be a converted image; and a drawing unit that draws an image in which the analysis results by the inference model are reflected in the surgical image.The image quality converter comprises an inverse gamma processing means that converts the output image into a linear signal, a means for performing image quality conversion on the linear signal including at least one of color conversion, brightness conversion, noise reduction, and edge enhancement, and a gamma processing means that performs gamma processing on the converted image.

[0008] According to the present invention, it is possible to provide a surgical image processing device that converts the image quality of a captured image into a desired image quality.

[0009] 1 is a diagram showing an overview of the functional configuration of a surgery support system to which the surgical image processing device of the present embodiment is applied. A diagram showing a modified example of the image quality converter 20. A diagram showing an overview of conventional image quality conversion. A diagram showing an overview of image quality conversion of the present embodiment. A diagram showing the processing flow in a surgery support system. A diagram showing a conventional method of estimating inverse gamma. A diagram showing a method of estimating inverse gamma of the present embodiment. A diagram showing a white balance checking jig of the present embodiment. A diagram showing a state in which the white balance checking jig is separated. A diagram showing a white balance checking jig of another embodiment. A diagram showing an example of conventional exposure adjustment. A diagram showing a method of determining a fixed exposure compensation ratio. A diagram showing an example of exposure compensation. A diagram showing selection of a conventional image quality conversion formula. A diagram showing selection of an image quality conversion formula of the present embodiment. A diagram showing an example of conventional image quality conversion. A diagram showing steps in conventional conversion from image quality A-3 to image quality B-2. A diagram showing an example of image quality conversion of the present embodiment. A diagram showing steps in conversion from image quality A-3 to image quality B-2 of the present embodiment. A diagram showing an example of conventional image quality conversion when there are three endoscopes. A diagram showing an example of image quality conversion of the present embodiment when there are three endoscopes.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes an embodiment of the present invention with reference to the accompanying drawings. Fig. 1A is a diagram showing an outline of the functional configuration of a surgery assistance system 100. The surgery assistance system 100 is equipped with a surgery image processing device 1.

[0011] (Surgery Support System) The surgery support system 100 is a system that supports a surgeon in surgery. Images captured of the surgeon's surgery are called surgical images. The surgical images are processed in various ways in the surgery support system 100. The surgery support system 100 supports the surgeon in surgery by displaying the processed images.

[0012] The surgery support system 100 includes a camera 102, a display device 104, and a surgical image processing device 1. The camera 102 is a device that acquires surgical images. An example of the camera 102 is an endoscopic camera. Note that the camera 102 is not limited to an endoscopic camera, as long as it is a camera that captures images of the surgical situation. The camera 102 may be, for example, a camera installed in an operating room or a camera attached to a surgical support robot. Furthermore, the camera 102 may be a camera that uses invisible light, hyperspectral imaging, or the like in addition to visible light. The display device 104 is a device that displays an output image from the display unit 16. Examples of the display device are a console and an LCD.

[0013] (Surgical image processing device) The surgical image processing device 1 is a device that determines the display content, etc., of the surgical assistance system 100. The surgical image processing device 1 includes a pre-processing unit 10, an inference unit 12, and a drawing unit 14. The surgical image processing device 1 may also include a display unit 16. The pre-processing unit 10, the inference unit 12, the drawing unit 14, and the display unit 16 process the surgical images acquired by the camera 102.

[0014] (Inference Unit) An AI model that analyzes surgical images and infers anatomical structures, instrument trajectories, events such as bleeding, surgical procedures, surgical scenes, and the state of instruments relative to anatomical structures is called an inference model. The inference unit 12 includes an inference model that is applied to the surgery assistance system 100.

[0015] (Drawing Unit) A surgical image reflecting the analysis results of the inference unit 12 is called a drawn image. The manner in which the analysis results of the inference unit 12 are reflected in the drawn image is called a drawing manner. The drawing unit 14 draws the drawn image in accordance with a predetermined drawing manner.

[0016] (Display Unit) The manner in which a drawn image is displayed on a display device is called the display mode. The display unit 16 displays the drawn image in accordance with a predetermined display mode. The drawn image displayed by the display unit 16 may be an image whose image quality has been converted by the image quality converter 20 of the pre-processing unit 10, which will be described below, or may be an image in its pre-image quality conversion state. In the latter case, the inference unit 12 performs inference using the image whose image quality has been converted, and when the image is subsequently displayed on the display unit 16, the image in its pre-image quality conversion state is used. The display unit 16 may be a display separate from the surgical image processing device 1, or may be displayed on a display attached to the display unit 16 or the surgical support robot.

[0017] (Pre-processing section) The surgical image processing apparatus 1 further includes a pre-processing section 10. The pre-processing section 10 will be described below.

[0018] Surgical images are acquired by a camera 102 at a medical institution or the like. Information indicating the state of the body is called body information. Information indicating the state of the instruments being operated by the surgeon is called instrument information. In the inference unit 12, the body information and instrument information contained in the surgical images are input into an inference model. In the inference model, the body information and instrument information are analyzed using AI. The inference model infers anatomical structures, instrument trajectories, and the state of the instruments relative to the anatomical structures.

[0019] The cameras 102 used in medical institutions are manufactured by various manufacturers. The models of the cameras 102 used in medical institutions are also various. The image quality of the acquired surgical images differs depending on the model of the camera 102. Examples of image quality include spatial frequency, brightness, and color tone.

[0020] The inference model is generated by learning from surgical images. The surgical images from which the inference model is learned are called training images. The training images are acquired by a specific camera 102.

[0021] The surgical images used in the surgical image processing device 1 are called images for inference. More specifically, the surgical images input to the inference model are called images for inference. The camera that captured the training images may be different from the camera 102 that captured the images for inference.

[0022] In this case, the image quality of the training image may differ from the image quality of the image for inference. If the image quality of the training image differs from the image quality of the image for inference, the accuracy of the analysis results in the inference unit 12 may decrease.

[0023] The pre-processing unit 10 reduces the difference in image quality between the training images and the images for inference. By reducing the difference in image quality between the training images and the images for inference, the pre-processing unit 10 suppresses a decrease in the accuracy of the analysis results of the inference unit 12.

[0024] The preprocessing unit 10 converts the image quality of the image for inference to that of the training image. The conversion is performed using a conversion formula. The conversion formula converts the image quality of the image for inference to an image quality that approximates the image quality of the training image. Furthermore, the training image may be converted to the image for inference, and the subject of the image does not matter as long as the method reduces the difference in image quality.

[0025] The conversion formula is set in the pre-processing unit 10. A person who uses the surgery assistance system 100 is called a user. The conversion formula is set by the user. Alternatively, the conversion formula may be set automatically.

[0026] The image transformed in the preprocessing unit 10 is called a transformed image. The transformed image is input to the inference unit 12.

[0027] The conversion formula may be a formula for converting the image quality of the inference image into an image quality that improves the analysis accuracy of the inference unit 12. The content of the conversion formula is not limited. The conversion formula may be a formula for performing frequency analysis on the inference image and converting the image using a frequency approach.

[0028] The database in which the conversion formula is stored is called the image quality converter 20. The image quality converter 20 is included in the pre-processing unit 10. The image quality converter 20 may be stored in the storage means of the surgery assistance system 100. The image quality converter 20 may also be downloaded from the cloud, a server, or the like.

[0029] An overview of image quality conversion will be described. FIG. 2 is a diagram showing an overview of conventional image quality conversion. FIG. 3 is a diagram showing an overview of image quality conversion of this embodiment. (Output Image) The image output by the camera 102 is called the output image 110. The image acquired by the camera 102 is usually subjected to gamma processing before being output from the camera 102. When gamma processing is applied to the image, the tone response characteristics of the image become nonlinear. The tone response characteristics of the output image 110 are usually nonlinear.

[0030] Hereinafter, when the tone response characteristics of an image are nonlinear, it will be referred to as "the image signal is nonlinear," etc. Similarly, when the tone response characteristics of an image are linear, it will be referred to as "the image signal is linear," etc.

[0031] (Image Quality Conversion) As shown in FIG. 2, when the image signal of the output image 110 is nonlinear, image quality conversion is performed on the nonlinear signal.

[0032] An example will be described in which the image quality conversion is color conversion. The image after the image quality conversion is called a converted image 112. The color conversion is performed by performing a matrix operation on each of the R, G, and B signals. The matrix operation is performed using a conversion matrix. Alternatively, the color conversion is performed by adjusting the intensity of each of the R, G, and B signals.

[0033] When a nonlinear signal is color-converted, the color is converted to a different color depending on the brightness. As a result, the image quality of the output image cannot be converted to an image quality that is close to that of the training image. The difference between the image quality of the converted image 112 and that of the training image becomes large.

[0034] Conversion of image brightness is similar to conversion of image color. Brightness correction is performed by multiplying the image signal by a gain. When converting a nonlinear signal, the weight of the gain differs depending on the brightness. As a result, the image quality of the output image 110 cannot be converted to an image quality that is close to the image quality of the training image. As with color and brightness, when a nonlinear signal is converted, the intended image quality conversion results cannot be obtained for contrast and sharpness.

[0035] In this embodiment, as shown in Fig. 3, inverse gamma processing is performed on the output image 110. The nonlinear signal of the output image 110 is converted into a linear signal by the inverse gamma processing. Once the output image 110 has become a linear signal, the linear signal is subjected to image quality conversion to obtain the intended image quality conversion result.

[0036] Next, the color-converted image is subjected to the same gamma processing as that of the training image, thereby converting the output image 110 into a converted image 112 having the image quality of the training image.

[0037] By converting the linear signal into a different image quality, it is possible to obtain image quality equivalent to that of the target camera, such as the camera that captured the surgical images used by the inference model, even if the brightness is different.

[0038] The processing flow in the surgery assistance system 100 will be described in more detail with reference to Fig. 4. Fig. 4 is a diagram showing the processing flow in the surgery assistance system 100. The following description will focus on the processing in the image quality converter 20.

[0039] (Source Camera) The camera 102 is the source camera. An example of the camera 102 is an endoscopic camera. Signal processing in the camera 102 will be described. The camera 102 includes an Image Sensor IS. The Image Sensor IS acquires surgical images. The image signal of the image acquired by the Image Sensor IS is a linear signal. The image signal is configured in a Bayer array of R, Gr, B, and Gb.

[0040] (Linear processing (front stage)) The image signal acquired by the Image Sensor IS undergoes a series of signal processing steps included in Linear processing (front stage) 107. Linear processing (front stage) 107 includes processing steps A-1 to A-5 and C-1. The configuration of the image signal is converted by undergoing each step of processing in Linear processing (front stage) 107. The configuration of the image signal changes from Bayer array data of R, Gr, B, Gb to data separated into R, G, B.

[0041] The linear processing (front stage) 107 includes BLC (Black Level Correction) A-1, Sensor correction (Defect pixel / Shading / Linearity) A-2, Lens correction (Shading / chromatic aberration of magnification) A-3, RAW-NR (Noise Reduction) A-4, White Balance A-5, and DeMosaic C-1. The image signal acquired by the Image Sensor IS undergoes the above-mentioned processes in the above-mentioned order.

[0042] (Linear processing (post-stage)) The image signal converted into data separated into R, G, and B in the linear processing (pre-stage) 107 is further processed in the linear processing (post-stage) 108. In the linear processing (post-stage) 108, the image signal is converted from a linear signal to a nonlinear signal.

[0043] The linear processing (post-stage) 108 includes a CCM (Color Correction Matrix) color conversion A-6 and a gamma (γ) processing C-2. The image signal converted into data separated into R, G, and B undergoes the above-mentioned processing in the above-mentioned order. The image signal becomes a nonlinear signal by undergoing the gamma processing C-2.

[0044] (Non-Linear Processing) The image signal converted into a non-linear signal in Linear processing (later stage) 108 is further processed in Non-Linear processing (non-linear processing) 109. The image signal is converted from the RGB format to the YCbCr format in Non-Linear processing 109. The image signal that has been subjected to Non-Linear processing 109 is output from camera 102.

[0045] Non-Linear Processing 109 includes RGB-NR (noise reduction) A-8, RGB2YC (RGB to YCbCr) C-3, luminance correction and color difference correction A-9, edge emphasis A-10, and YC-NR (noise reduction) A-11. The non-linear signal of the RGB system undergoes the above-mentioned processes in the above-mentioned order.

[0046] The image that has undergone the above processing is output from the camera 102 as an output image. The signal output from the camera 102 is called an output signal. The RGB nonlinear signal is converted into a YCbCr nonlinear signal by the RGB2YC processing C-3. The image signal of the output image is a YCbCr nonlinear signal.

[0047] (Image Quality Converter) The following describes the processing in the image quality converter 20. The processing in the image quality converter 20 includes processing 21 for converting the output signal, which is a nonlinear signal, into a linear signal, and processing 22 for converting the image quality of the output image into the image quality of the destination camera. The conversion processing 21 into a linear signal is performed by performing an inverse operation of the processing in the source camera.

[0048] (Conversion Process to Linear Signal) The conversion process to linear signal 21 will be described. The conversion process to linear signal 21 includes YC2RGB (YCbCr system to RGB system) C-3', inverse edge emphasis A-10', inverse luminance correction and inverse color difference correction A-9', and γ －１ and an inverse gamma C-2' (inverse gamma processing means). These processes are the inverse operations of the corresponding processes in the linear processing (post-stage) 108 and non-linear processing 109. The symbols for the processes in the image quality converter 20 are the symbols obtained by adding (') to the symbols for the corresponding processes in the linear processing (post-stage) 108 and non-linear processing 109.

[0049] The YCbCr nonlinear signal input to the image quality converter 20 is converted into an RGB linear signal in a linear signal conversion process 21. The RGB linear signal undergoes offset correction E-1 (offset adjustment), exposure correction (gain correction) E-2 (brightness adjustment), and white balance correction E-3 (white balance correction).

[0050] The image signal that has been processed from E-1 to E-3 is subjected to inverse CCM color conversion A-6', which is the inverse operation of the CCM color conversion A-6 in the linear processing (previous stage) 107. The inverse CCM color conversion A-6' can be omitted.

[0051] (Processing for conversion to image quality of destination camera) The following describes processing for conversion to image quality of destination camera 22. Processing for conversion to image quality of destination camera 22 is similar to the processing included in linear processing (later stage) 108 and non-linear processing 109.

[0052] The conversion process 22 to the image quality of the destination camera includes CCM color conversion B-6, γ (Gamma) processing D-2 (gamma processing means), RGB-NR B-8 (noise reduction), RGB2YC D-3, brightness correction and color difference correction B-9 (color conversion, brightness conversion), edge emphasis B-10, and YC-NR B-11. The image signal undergoes the above processes in the above order.

[0053] The purpose of CCM color conversion B-6 in conversion processing 22 to image quality of the destination camera is similar to the purpose of CCM color conversion A-6 in linear processing (later stage) 108. However, CCM color conversion B-6 has conversion content corresponding to the destination camera.

[0054] As for other processes, the purpose of processes with the same name is the same between the process included in the conversion process 22 to the image quality of the destination camera and the processes included in the linear processing (later stage) 108 and non-linear processing 109. However, the specific content of the process included in the conversion process 22 to the image quality of the destination camera has conversion content corresponding to the destination camera.

[0055] RGB-NR B-8 and YC-NR B-11 can be omitted.

[0056] The image that has undergone conversion processing 22 to the image quality of the destination camera is output from the image quality converter 20. The image output from the image quality converter 20 is called a converted image. The image signal of the converted image is a non-linear signal in the YCbCr format.

[0057] In this embodiment, the processing in the image quality converter 20 includes a conversion process 21 to a linear signal. Then, the linear signal that has undergone the conversion process 21 to a linear signal is subjected to a conversion process 22 to the image quality of the destination camera. Therefore, the image quality of the converted image can be converted to an image quality that approximates the image quality of the training image.

[0058] As described above, in the surgical image processing device 1, the output image from the camera 102 is subjected to image quality conversion processing in the image quality converter 20. The output image is converted into the image quality of the surgical image learned by the inference model through the image quality conversion processing in the image quality converter 20. The image quality conversion processing in the image quality converter 20 includes conversion processing to a linear signal, offset correction, gain correction, white balance correction, and conversion processing to the image quality of the destination camera.

[0059] The above-described signal processing means and signal processing order performed by the image quality converter 20 are merely examples. The signal processing means and signal processing order within the source camera 102 differ depending on the camera 102. Therefore, it is preferable to change the type, means, order, etc. of signal processing required for linear signal conversion depending on the camera 102.

[0060] Similarly, with regard to the conversion process to the image quality of the destination camera, it is preferable to change the type, means, and order of signal processing depending on the destination camera.

[0061] In the above description, the image quality conversion process is performed on RGB signals (RGB format) and YC signals (YCbCr format). The image quality conversion process may be performed on all RGB signals. Furthermore, the image quality conversion process may be performed on color space signals other than RGB signals and YC signals.

[0062] The image quality conversion process may be realized by software or by hardware such as a signal processing LSI.

[0063] (Method of estimating inverse gamma) γ －１ The method of estimating inverse gamma in inverse gamma C-2' will be described below. Fig. 5 is a diagram showing a conventional method of estimating inverse gamma. Fig. 6 is a diagram showing the method of estimating inverse gamma according to this embodiment.

[0064] The signal level of the endoscopically captured image 120 and the absolute value of the sensor output are required to estimate the gamma curve from the endoscopically captured image 120. The absolute value of the sensor output may be a ratio to saturation.

[0065] When the absolute value of the sensor output is unknown, a graph 130 is created in which the reflectance of each patch 125 of the gradation chart 124 is used as an input and the signal level of the endoscopic image 120 is used as an output.

[0066] The density of each patch 125 of the gradation chart 124 is predetermined. The density may be a reflectance. The gradation chart 124 may be a reflective or transmissive type. The light source is a light source whose brightness can be changed, or a brightness box.

[0067] The endoscopically captured image 120 is captured at a reference brightness. The signal level of the endoscopically captured image 120 is the average value of the green color within the frame 122 of each patch 121 in the endoscopically captured image 120.

[0068] Graph 130 is a diagram showing a gamma curve 131. The X-axis of graph 130 represents the reflectance of each patch 125 of the gradation chart 124 as an input. The Y-axis of graph 130 represents the signal level of each gradation in each patch 121 of the endoscopic image 120 as an output.

[0069] In the graph 130, values ​​are plotted for 24 patches 121 contained in one endoscopically captured image 120.

[0070] As shown in graph 130, a portion of a gamma curve can be generated using a conventional inverse gamma estimation method. However, the reflectance used as input is not the absolute value (relative to saturation) of the sensor output. Therefore, the plot shown in graph 130 is not a curve that matches the input and output. Therefore, a gamma curve cannot be estimated using the plot shown in graph 130.

[0071] The inverse gamma estimation method of this embodiment will be described with reference to Fig. 6. In the following description, differences from the conventional inverse gamma estimation method shown in Fig. 5 will be explained.

[0072] In the conventional inverse gamma estimation method, the endoscopic image 120 is captured at a standard brightness. In contrast, in the inverse gamma estimation method of this embodiment, the endoscopic image 120 is captured by sweeping the brightness. The sweep range is, for example, from ×0.25 to ×2.

[0073] Furthermore, in conventional inverse gamma estimation methods, the input is the reflectance of each patch 125 of the gradation chart 124. In contrast, in the inverse gamma estimation method of this embodiment, the input is a value obtained by multiplying the reflectance of each patch 125 of the gradation chart 124 by the brightness sweep ratio in endoscopic imaging. Intersection 127 in Fig. 6 indicates that the reflectance of each patch 125 of the gradation chart 124 is multiplied by the brightness sweep ratio in endoscopic imaging.

[0074] A graph 133 in FIG. 6 is a graph showing a gamma curve 134 obtained by the inverse gamma estimation method of this embodiment.

[0075] In the inverse gamma estimation method of this embodiment, endoscopic images are captured while sweeping the brightness. The input is the reflectance x brightness sweep ratio. Graph 130 plots the values ​​of 24 patches 121 for each number of images after sweeping the brightness. As described above, in the inverse gamma estimation method of this embodiment, the signal levels of all patches in all endoscopic images are plotted, and the point at which the captured image (output side) saturates is accurately determined.

[0076] The point at which the captured image (output side) saturates is indicated by point 136. The point at which the output saturates is designated as the input saturation point 136. A gamma curve 134 is created based on this saturation point 136. A frame 137 is drawn based on the saturation point 136 and an output value 138 when the input is the value at the saturation point 136. The gamma curve 134 is a graph normalized with the Max of the X axis of the frame 137 set to 1.0. By defining point 136 as the input multiplied by 1 (= saturation point), the absolute value of the input side (sensor saturation ratio) can be determined. This allows the gamma curve 134 to be estimated.

[0077] Furthermore, by capturing an endoscopic image while sweeping the brightness and plotting many coordinates, intermediate chart gradations can be interpolated, thereby improving the estimation accuracy of the gamma curve 134.

[0078] (Image Quality Converter) In the image quality converter 20 described with reference to Fig. 4, more accurate image quality conversion can be performed by appropriately adjusting the following items. Note that the following are just examples, and other adjustment items may be provided. Menu items Gain setting: 0 dB to Max dB White balance: G / R, G / B Contrast: High / Standard / Low, etc. Color tone adjustment: Step of R emphasis, B emphasis (e.g., 5 out of 10) Saturation adjustment: Strong / Standard / Weak, etc. Edge emphasis: Strong / Standard / Weak, etc. Noise reduction: Strong / Standard / Weak, etc.

[0079] The signal processing flow can be exemplified by the following items: Signal processing flow Gamma curve: For each contrast CCM matrix: For each gamma curve Color correction formula after gamma: For each color tone adjustment and saturation adjustment

[0080] The menu items are examples of common contents that are not dependent on the camera model, etc. You can check in advance how each menu item will change depending on the item, so if you know the current setting value, it will be easy to change the image quality.

[0081] The signal processing flow is illustrated mainly with respect to color reproduction, taking into consideration the degree of influence on inference. Regarding the signal processing flow, if setting information linked to gamma is disclosed, conversion will be easier.

[0082] With this information about the camera, a more appropriate image quality converter can be created.

[0083] (White Balance) White balance correction E-3 will now be described. In the image quality conversion formula for converting image quality from the source camera 102 to the destination camera, the conversion process to a linear signal includes a conversion formula that can be fixed by the camera and a conversion formula that needs to be variable depending on the shooting conditions of the source camera 102.

[0084] The latter conversion formulas that need to be variable include a conversion formula for white balance correction, a conversion formula for exposure correction that includes gain multiplication, and conversion formulas related to these conversion formulas. In addition, edge emphasis linked to AF is also included in the conversion formulas that need to be variable.

[0085] The white balance may be set manually by the user or the default settings may be used as they are. When the white balance is set manually, the white balance may not be appropriate. For example, the white balance may not be appropriate due to the influence of external light.

[0086] The white balance set at the time of shipment from the manufacturer may not necessarily be the appropriate white balance. Due to individual variations in cameras and changes in parts over time, the appropriate white balance may not be achieved.

[0087] If the image quality is converted with an inappropriate white balance, color shifts will occur not only in white subjects but also in subjects with low saturation, making it impossible to convert the image quality of the output image to a quality that is close to the image quality of the surgical images that the inference model learned.

[0088] Unlike endoscopic cameras, ordinary cameras use external light for white balancing. Endoscopic cameras capture images of the inside of the body using light emitted from the tip of the camera. Endoscopic cameras have their own light source. Therefore, white balancing must be performed using the endoscope's light source.

[0089] The method of correcting the white balance correction value in this embodiment uses a white balance checking jig 150. The white balance checking jig 150 will be described. Fig. 7 is a diagram showing the white balance checking jig 150 of this embodiment. A cross-sectional view 158 of the white balance checking jig 150 is also shown in Fig. 7.

[0090] 7, the white balance checking jig 150 includes a cylindrical jig body 151. The shape of the jig body 151 may be a rectangular parallelepiped, a cone, or the like.

[0091] An endoscope insertion port 156 is formed on one bottom surface 152 of the jig body 151. The endoscope insertion port 156 is a through-hole for inserting the tip of the endoscopic camera 102 into the jig body 151. The endoscope insertion port 156 may be shielded from light by a light-shielding ring or the like.

[0092] An achromatic color chart 160 (achromatic subject) is placed on the other bottom surface 154 inside the jig body 151. The achromatic color chart 160 is a member that is colored gray with a predetermined brightness.

[0093] The white balance checking jig 150 can be separated into multiple parts. Figure 8 shows the white balance checking jig 150 in a separated state. The jig body 151 is separated into a tip chart part 168, a housing part 166, and an endoscope insertion port part 164.

[0094] The leading edge chart portion 168 including the achromatic chart 160 can be removed from the housing portion 166 in the same way as removing a cap, making it easy to clean the inside of the jig body 151.

[0095] The endoscope insertion port 164, which includes the endoscope insertion port 156, can also be detached from the housing 166. Therefore, it is possible to replace the endoscope insertion port 164 with a different endoscope insertion port 156 depending on the manufacturer, model, tip diameter, shape, etc. of the camera 102 (endoscopic camera).

[0096] Taking into consideration autoclave resistance and light-blocking properties, the exterior of each part can be made of metal materials such as aluminum, CFRP (carbon fiber reinforced plastic), light-blocking polypropylene, etc. The achromatic color chart 160 can be made by coloring the inside of a polypropylene sheet gray.

[0097] The method for correcting the white balance correction value in this embodiment uses a white balance checking jig 150 that is attached to the tip of the camera 102 (endoscopic camera), can completely block light, and has a certain area or more in the center of the angle of view that is achromatic (gray). The tip of the camera 102 is inserted into the jig body 151 through the endoscope insertion port 156. The white balance is checked using an achromatic chart 160. Specifically, a white balance error is measured, and the white balance correction value of the image quality converter 20 is corrected based on that error. The image quality converter 20 may include a white balance error generating means that generates a white balance error.

[0098] The white balance check is started in response to an instruction from the user or when the camera 102 automatically determines from an image that the white balance check tool 150 is attached.

[0099] (Low saturation pixels, low saturation regions) Another embodiment is a method that does not use the white balance checking jig 150. In this method, the white balance error is estimated based on the R / G and B / G signal ratios of a subject with low saturation and chromatic colors in the surgical field image. The white balance correction value of the image quality converter 20 is corrected based on the estimated error.

[0100] Specifically, it determines low saturation pixels, which are pixels of chromatic colors with low saturation in the output image 110, or low saturation regions, which are regions of chromatic colors with low saturation in the output image 110. A low saturation determination means for making this determination may be provided in the image quality converter 20.

[0101] Next, a white balance gain for the low saturation pixels or a white balance gain for the low saturation region is generated. The white balance gain generating means for generating this white balance gain may be provided in the image quality converter 20.

[0102] Then, a white balance error for a low saturation pixel or a white balance error for a region is generated according to the color temperature. The white balance error generating means for generating this white balance error may be provided in the image quality converter 20.

[0103] The image quality converter 20 corrects the white balance correction value with the generated white balance error.

[0104] As described above, instead of detecting specific parts of the surgical field, the image is converted to a color space or equivalent color gamut, and white balancing is performed by focusing on low-saturation areas in that space. Color correction processing can also be performed using the image of the surgical field.

[0105] (Skin Color Check Chart) In another embodiment, there is a method of correcting the white balance correction value using a white balance check jig 150 equipped with a skin color check chart 162 instead of the achromatic color chart 160. Differences from the embodiment shown in FIG. 8 will be described. FIG. 9 is a diagram showing a white balance check jig 150 of another embodiment. As shown in FIG. 9, a skin color check chart 162 is arranged on the bottom surface 154 inside a jig main body 151. The skin color check chart 162 is a chart on which skin colors of different lightness, hue angle, and saturation in multiple tones are arranged.

[0106] When using a white balance check jig 150 (skin color reproduction check jig) equipped with a skin color check chart 162, an error in skin color reproduction is measured using the skin color check chart 162. A skin color reproduction error generating means for measuring the skin color reproduction error may be provided in the image quality converter 20. Based on the measured error, the white balance correction value of the image quality converter 20 and the color conversion process are corrected.

[0107] In each of the above-described embodiments, the operator may be notified of deviations in the white balance setting, deviations due to aging, etc. based on the obtained information, thereby supporting the use of the endoscopic camera in an appropriate state.

[0108] (Exposure Compensation) Exposure compensation E-2 will now be described. The exposure compensation conversion formula basically uses an exposure compensation ratio that is fixed for the combination of the source and destination cameras, which is the ratio of the exposure convergence levels of the source and destination cameras. The exposure convergence level is the brightness level at which the linear domain signal becomes the target.

[0109] However, for example, in the case of cameras equipped with automatic exposure or automatic brightness adjustment, the brightness of the image will vary from camera to camera depending on the conditions of the subject. For example, in a scene with a large difference in brightness, the brightness will vary from camera to camera depending on the size of the main subject and the magnitude of the difference in brightness.

[0110] Therefore, when converting the image quality of the output image from the source camera, if exposure correction is performed on the image converted into a linear signal using only a fixed exposure correction ratio that corresponds to the source and destination cameras, the image may be too bright or too dark depending on the gamma characteristics of the destination camera and the subject conditions, and the image cannot be converted to an image quality that approximates the image quality of the surgical image learned by the inference model.

[0111] An example of exposure compensation will be described with reference to FIG. 10 . FIG. 10 is a diagram illustrating an example of conventional exposure compensation. FIG. 10 shows an example in which an image is too bright after image quality conversion. Image 200 shows the image before conversion. Image 202 shows the image after conversion. Image 200 has a large difference in brightness within the image. Image 200 is an example in which dark areas 204 are enhanced and bright areas 206 are reproduced to the best of their ability. First, the image signal of this image 200 is converted into a linear signal. This conversion is performed by performing a process that creates an inverse gamma of the source gamma. Exposure compensation E-2 is then performed on the resulting linear signal. The exposure compensation ratio is the exposure convergence level of the destination image divided by the exposure convergence level of the source image. The exposure compensation ratio will be explained later. After exposure compensation E-2, gamma processing of the destination image is performed to obtain a converted image 202.

[0112] 10, the gamma of the destination image is significantly increased, so the signal level is lowered by exposure compensation. However, even if the exposure level is lowered by exposure compensation, the source image has a large difference in brightness, making the bright area 206 brighter, so the gradation of the bright area 206 cannot be reproduced with a fixed exposure compensation ratio. In endoscopic surgery, light is directed at the area that needs to be seen, so the bright side is more of a problem than the dark side.

[0113] Before describing exposure compensation according to this embodiment, a method for determining a fixed exposure compensation ratio will be described with reference to Fig. 11. Fig. 11 is a diagram showing how to determine a fixed exposure compensation ratio. In the example shown in Fig. 11, the brightness of the output image is set to be the same between the source camera and the destination camera. The brightness of the output image is 140 levels out of 256 levels.

[0114] First, a gray chart 210 with a reflectance of 18% is photographed with a source camera and a destination camera. The source camera and the destination camera have different exposure convergence levels. An image 212 taken with the source camera has a bright exposure convergence level. On the other hand, an image 216 taken with the destination camera has a dark exposure convergence level. Therefore, exposure correction is performed by setting the exposure correction ratio to the destination exposure convergence level / the source exposure convergence level. Then, gamma processing is performed on each image. This converts the linear signal output from the sensor into a nonlinear signal. With this exposure correction, the output images 214 and 218 from the cameras have the same brightness.

[0115] Exposure correction according to this embodiment will now be described. Fig. 12 is a diagram showing an example of exposure correction. In the following explanation, differences from the matters explained with reference to Fig. 10 will be explained.

[0116] For the exposure compensation conversion formula of the image quality converter 20, the maximum and minimum signal levels at which the converted signal will be at an appropriate level for the destination gamma are calculated in advance. An appropriate level is a level at which bright areas are not too bright to be reproduced, and dark areas are not too dark to be reproduced. The range from the minimum signal level to the maximum signal level at the appropriate level is called the predetermined level range. This predetermined level range can also be called the optimal level range. If, in the source image, signals greater than (or equal to) this maximum signal level exceed a predetermined tolerance, or if signals smaller than (or equal to) the minimum signal level exceed a predetermined tolerance, the exposure compensation ratio in the image quality converter 20 is corrected.

[0117] Specifically, as shown in the flow chart of Figure 12, a first preset 220 predetermines the maximum and minimum input levels at which the converted signal will be appropriate (input level calculation means). A second preset 222 predetermines the tolerance rate for pixels above the maximum value / below the minimum value (tolerance value setting means). A histogram 224 of the signal of the pre-conversion image 200 is calculated to count the number of pixels above the maximum value or below the minimum value determined in the first preset 220 (pixel ratio calculation means). The example shown in Figure 12 illustrates the case where the frequency of pixels above the maximum value is calculated. A box 226 indicates the frequency of pixels that will be too bright after conversion.

[0118] Then, in decision step 228, it is determined (comparison means) whether the frequency of pixels that are too bright after conversion, shown in box 226, exceeds the tolerance rate determined in second preset 222. If the determination is yes, the correction rate is reduced, i.e., the gain is reduced.

[0119] The same applies to the minimum value defined in the first preset 220. If the frequency of pixels that are too dark after conversion exceeds the tolerance defined in the second preset 222, the correction ratio is increased, i.e., the gain is increased.

[0120] An endoscopic camera shines light on the area to be viewed, i.e., the area where surgery is performed. Therefore, it is necessary to suppress blown-out highlights and color shifts on the brighter side rather than the darker side. Implementing AE (auto exposure) in the image quality converter 20 requires a great deal of effort, and in conditions where there is no difference in brightness, the AE on the camera itself is sufficient. Therefore, incorporating AE in the image quality converter 20 can be said to be overkill.

[0121] As described above, exposure correction is performed at a fixed ratio according to the camera's AE, and if the image becomes too bright due to the gamma of the destination camera, the fixed ratio can be corrected to obtain optimal exposure with little cost and resources.

[0122] (Selection of conversion formula) The conversion formula for image quality conversion, i.e., the selection of the image quality conversion formula, will be explained. The image quality conversion formula differs depending on the camera used. Therefore, the conversion formula is set by the user from a database that stores a series of conversion formulas for converting to the image quality (spatial frequency, brightness, color tone, etc.) equivalent to the destination, or the conversion formula is automatically set by the inference model selected by the user. If the user sets the conversion formula incorrectly, the appropriate conversion formula will not be selected. As a result, the inference model will not be able to convert to an image quality that is close to the image quality of the surgical image it has learned.

[0123] In this embodiment, the image quality conversion formula is selected by analyzing the characteristics (color distribution, histogram, spatial frequency, etc.) of the image output from the camera and pre-storing the characteristics in a database or the like that stores conversion formulas. Then, when selecting an image quality conversion formula, the system determines which pre-stored camera the characteristics of the image output from the camera match, and automatically selects a conversion formula registered in the database. Alternatively, if the conversion formula selected by this means differs from the conversion formula set by the user or automatically set by the inference model selected by the user, a warning message is displayed to prompt the user to confirm whether the settings are appropriate. Image quality characteristics can be registered not only for each camera, but also for each menu setting for each camera.

[0124] The following description will be given with reference to the drawings. Fig. 13 is a diagram showing the selection of a conventional image quality conversion formula. Fig. 14 is a diagram showing the selection of an image quality conversion formula of this embodiment. In Figs. 13 and 14, an image 230 indicates the image to be converted, and an image 232 indicates the image to be converted.

[0125] As shown in Fig. 13, conventionally, the conversion formula applied to the image quality converter 20 is selected by the user from among the conversion formulas stored in the image quality conversion formula DB 234, or is automatically selected from an inference model. In the example shown in Fig. 13, the source camera is camera A, the destination camera is camera B, and the conversion formula A-B-1 is selected. Here, it is possible that the selection of the conversion formula A-B-1 is incorrect.

[0126] In contrast, in the selection of an image quality conversion formula in this embodiment, as shown in FIG. 14 , a conversion formula is automatically selected by referencing image quality feature information stored in advance in the image quality conversion formula DB 234. Examples of image quality feature information are histograms and color space mapping. When selecting an image quality conversion formula, image quality features are generated based on information from the image quality converter 20, etc. (analysis means). Examples of image quality features are preferably the same as the image quality feature information. Examples of image quality features are histograms and color space mapping. Based on the image quality features acquired from the image quality converter 20, the image quality feature information stored in the image quality conversion formula DB is referenced (determination means), and a conversion formula is automatically selected. This enables the selection of a conversion formula with fewer errors.

[0127] We will explain in more detail about selecting the image quality conversion formula. The image quality of a camera varies depending on the image sensor, signal processing LSI, image sensor and signal processing LSI settings, auto exposure convergence level, and white balance, and the final result varies from manufacturer to manufacturer. Furthermore, even the same endoscope system can produce different image quality depending on the menu settings.

[0128] On the other hand, the image quality itself can be evaluated by analyzing the image using indicators such as color reproduction, brightness, contrast, tone reproduction, resolution, edge strength, and noise, and different evaluation values ​​can be obtained for each of these image qualities.

[0129] For example, with regard to color reproduction, by plotting an image in a color space such as L*a*b* and quantifying the center position and distribution trend, it is possible to estimate the camera type and menu setting candidates. In the case of surgery, the organs that will be included in the surgical field are determined by the surgical area and surgical procedure, so the judgment index may be changed depending on the surgical target. Furthermore, the color space state of the surgical field changes depending on the patient's condition (e.g., excessive fat) and the intraoperative situation (e.g., excessive bleeding), so the judgment index and image quality conversion method may be changed accordingly.

[0130] In addition, for example, by estimating candidate camera and menu settings from numerical values ​​of contrast and tone reproduction characteristics obtained from the image histogram and judging them together with the estimated results based on color reproduction, it is possible to determine the camera type and menu settings with greater accuracy.

[0131] During surgery, the camera may be removed from the body to clean the lens. This may result in an image that differs from the actual surgical field. Therefore, the image quality characteristics may be judged only when the image matches the color conditions specific to the surgical field (organs, blood vessels, etc.). In addition, the judgment results may be stabilized over time by, for example, applying a low-pass filter to the judgment results.

[0132] The technical scope of the present disclosure is not limited to the above-described embodiments. The above-described embodiments may be variously modified and improved. Furthermore, the modified or improved forms are included in the technical scope of the present disclosure.

[0133] For example, the image quality converter 20 may have a means for performing image quality conversion including at least one of color conversion, brightness conversion, noise reduction, and edge enhancement on the nonlinear signal before converting the output image 110 into a linear signal, thereby enabling image quality conversion using the nonlinear signal in accordance with the configuration of the source camera.

[0134] Furthermore, conversion from a nonlinear signal to a linear signal and conversion from a linear signal to a nonlinear signal can be omitted as appropriate depending on the camera configuration, etc. This may improve conversion accuracy or inference accuracy.

[0135] In the above embodiment, each process is described using an example in which the image quality of the output image 110 is converted to an image quality that is similar to the image quality of the surgical image on which the inference model has been trained. Each process may also be used to generate a training image on which the inference model is trained.

[0136] An example of the process flow when using image quality conversion to generate training images is as follows. The following uses still images as an example, but the images can also be videos. "Acquire surgical still images --> convert the image quality of the acquired surgical still images --> add annotations (for example, correct answer data) to the surgical still images after image quality conversion --> use this data as training data to create an inference model through machine learning." Examples of annotations include various items related to the evaluation of the level of surgery, the body, and surgical instruments.

[0137] During learning, images or videos in which at least one of the source image quality and the destination image quality is different may be mixed for learning.

[0138] 1B, the image quality converter 20 may be provided separately from the surgery assistance system 100. The image quality converter 20 may also be provided with an instruction unit 31. The image quality converter 20 is then connected to a learning unit 32.

[0139] In the above embodiment, the surgical image processing device has been described as an object. However, the surgical image processing device may be a program that causes the surgical image processing device to function as each unit.

[0140] (Conversion to Image Quality of Other Endoscopes) The conversion of image quality in the image quality converter 20 described with reference to Fig. 4 etc. will be specifically described. First, the conventional technology will be described. Fig. 15 is a diagram showing an example of conventional image quality conversion. Fig. 15 shows an example of converting images of various image qualities of endoscope A into images of various image qualities of endoscope B.

[0141] Even images captured with the same endoscope may have different image quality. In the example shown in Fig. 15, endoscope A has three types of image quality: image quality A-1, image quality A-2, and image quality A-3. The image quality A-1, image quality A-2, and image quality A-3 differ in white balance and image quality settings made by the user. Of the image quality A-1, image quality A-2, and image quality A-3, image quality A-1 is set as the default image quality.

[0142] The same applies to endoscope B. Endoscope B has image quality B-1 and image quality B-2. Image quality B-1 and image quality B-2 have different white balances. Image quality B-1 is set as the default image quality.

[0143] When converting image quality from endoscope A to endoscope B, six types of image quality conversion routes are possible, from route 1 (R1) to route 6 (R6), as shown in Fig. 15. Route 1 (R1) is a route for converting from image quality A-1 to image quality B-1, route 2 (R2) is a route for converting from image quality A-1 to image quality B-2, route 3 (R3) is a route for converting from image quality A-2 to image quality B-1, route 4 (R4) is a route for converting from image quality A2 to image quality B-2, route 5 (R5) is a route for converting from image quality A-3 to image quality B-1, and route 6 (R6) is a route for converting from image quality A-3 to image quality B-2.

[0144] 16 is a diagram showing steps of a conventional conversion from image quality A-3 to image quality B-2. The conversion shown in FIG. 16 corresponds to the conversion of root 6 (R6) in FIG.

[0145] As shown in Figure 16, route 6 (R6) includes four steps: inverse gamma H-1, white balance adjustment H-2, color conversion H-3, and gamma H-4. The processing content of each step is the same as that described with reference to Figure 4, etc. When converting image quality from image quality A-3 to image quality B-2, first, in the inverse gamma H-1 step, the image is converted from non-linear to linear. Next, white balance adjustment H-2 and color conversion H-3 are performed. Furthermore, in the gamma H-4 step, the image is converted from linear to non-linear. As a result, image quality A-3 is converted to image quality B-2.

[0146] Of the four steps included in route 6 (R6), namely, inverse gamma H-1, white balance adjustment H-2, color conversion H-3, and gamma H-4, inverse gamma H-1 is common to image quality A, white balance adjustment H-2 and color conversion H-3 are dedicated to conversion from image quality A-1 to image quality B-2, and gamma H-4 is common to image quality B. "Common to image quality A" means that it can be used in common for various image qualities such as image quality A-1, image quality A-2, and image quality A-3 in endoscope A. Similarly, "common to image quality B" means that it can be used in common for various image qualities such as image quality B-1 and image quality B-2 in endoscope B.

[0147] In contrast, the white balance adjustment H-2 and color conversion H-3 are dedicated to individual conversions, because color conversion depends on white balance and therefore different processing is required for each white balance.

[0148] Each conversion route includes steps for performing different conversions for each combination of conversions, such as white balance adjustment H-2 and color conversion H-3. Therefore, a conversion formula for converting image quality is required for each route. In the example shown in Figure 15, six conversion formulas are required, one for each of the six routes, from route 1 (R1) to route 6 (R6).

[0149] In this way, when an image quality converter is used to support many endoscopes and endoscope systems, if conversion formulas are created individually for each endoscope, the number of conversion formulas becomes enormous.

[0150] The image quality converter 20 of this embodiment will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing an example of image quality conversion of this embodiment. The following description will focus on differences from the conventional image quality converter described with reference to Figures 15 and 16. Items not specifically described in the following description can be made the same as those in the conventional image quality converter.

[0151] As shown in Fig. 17, in the image quality converter 20 of this embodiment, image quality conversion is performed via the respective default image quality of endoscope A and endoscope B. Image quality other than the default image quality of the endoscope at the source of conversion is first converted to the default image quality. Then, the default image quality of the endoscope at the source of conversion is converted to the default image quality of the endoscope at the destination of conversion. Thereafter, as necessary, the image quality is converted to the non-default image quality of the endoscope at the destination of conversion.

[0152] 17, image quality A-2 is converted to image quality A-1 via route 11 (R11), image quality A-3 is converted to image quality A-1 via route 12 (R12), and image quality A-1 is converted to image quality B-1 via route 1 (R1).

[0153] When it is necessary to convert to image quality B-2, image quality B-1 is converted to image quality B-2 by route 21 (R21).

[0154] The image quality converter 20 of this embodiment does not directly convert from each source image quality to each destination image quality, but rather performs conversion via the default image quality of each of endoscope A and endoscope B. This makes it possible to reduce the number of types of conversion formulas that need to be prepared.

[0155] In the conventional example shown in Figure 15, there are six types of conversion routes, and therefore six types of conversion formulas must be prepared. In contrast, in the example of this embodiment shown in Figure 17, there are only four types of conversion routes, and therefore four types of conversion formulas must be prepared. Furthermore, as will be explained later, the same conversion formula can be used for root 11 (R11) and root 12 (R12). Therefore, in the example of this embodiment, only three types of conversion formulas must be prepared.

[0156] Furthermore, converting image quality between different endoscopes places a greater burden on the conversion process than converting image quality within the same endoscope. With the image quality converter 20 of this embodiment, image quality conversion between different endoscopes can be performed only using root 1 (R1). This further reduces the burden of preparing a conversion formula.

[0157] An example of image quality conversion in the image quality converter 20 of this embodiment will be described with reference to Fig. 18. Fig. 18 is a diagram showing the steps of conversion from image quality A-3 to image quality B-2 of this embodiment. The conversion shown in Fig. 18 corresponds to the conversion of root 12 (R12), root 1 (R1), and root 21 (R21) in Fig. 17.

[0158] As shown in Figure 18, Route 12 (R12) includes two steps: inverse gamma H-21 and white balance adjustment H-22. Route 1 (R1) includes color conversion H-23. Route 21 (R21) includes two steps: white balance adjustment H-24 and gamma H-25.

[0159] Image quality A-3 is converted to image quality A-1 through the steps included in route 12 (R12). Image quality A-1 is converted to image quality B-1 through color conversion H-23 included in route 1 (R1). Image quality B-1 is converted to image quality B-2 through the steps included in route 21 (R21).

[0160] Of the three routes involved in the conversion from image quality A-3 to image quality B-2, Route 12 (R12), Route 1 (R1), and Route 21 (R21), Route 12 (R12) is common to image quality A, Route 1 (R1) is exclusive to the conversion from image quality A-1 to image quality B-2, and Route 21 (R21) is common to image quality B.

[0161] As shown by the fact that root 12 (R12) is common to image quality A and root 21 (R21) is common to image quality B, conversion within the same endoscope can use the same conversion formula regardless of the image quality of the source and destination. In the example of endoscope A, the conversion formula for root 11 (R11) and the conversion formula for root 12 (R12) can be the same. This is because conversion processes other than color conversion do not depend on the image quality of the source and destination.

[0162] In the image quality converter 20 of this embodiment, by preparing a conversion formula within the endoscope at the source endoscope, a conversion formula from the default image quality at the source endoscope to the default image quality at the destination endoscope, and a conversion formula within the endoscope at the destination endoscope, it is possible to convert the image quality of the source endoscope to the image quality of the destination endoscope.

[0163] (Summary) Of the conversion processes performed by the image quality converter 20, color conversion requires conversion according to the image quality of the source and destination. Therefore, different conversions are required depending on the combination of the endoscope and its image quality. For example, different white balances require different color conversions. In other words, a conversion formula must be prepared for each combination of the source endoscope and destination endoscope.

[0164] On the other hand, in conversion processes other than color conversion, including white balance, the conversion formula does not have dependency on the endoscope of the source and the endoscope of the destination, so that independent conversion is possible for the source and the destination.

[0165] Therefore, in the source endoscope, regardless of the image quality, first, the image quality is converted to the default setting of the source endoscope, then converted to the default setting of the destination endoscope, particularly color conversion, and then converted from the default setting of the destination endoscope to the desired image quality.

[0166] As a result, the image quality converter 20 of this embodiment can suffice with a single conversion formula between endoscopes.

[0167] On the other hand, white balance conversion is performed on both the source endoscope and the destination endoscope. However, conversion processes other than color conversion are simpler than conversion processes for color conversion.

[0168] Furthermore, conversion processes other than color conversion are common processes within each endoscope and are not dependent on image quality, which prevents the number of conversion formulas from becoming too large.

[0169] (When there are three or more endoscopes) Conventional image quality conversion when there are three or more endoscopes will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example of conventional image quality conversion when there are three endoscopes. When converting the image quality of a source endoscope to the image quality of a destination endoscope, if the number of endoscopes to be supported increases, a new conversion formula must be created on the endoscope that was previously compatible. This is the same whether conversion is performed individually for each image quality as in the past, or conversion is performed via the default image quality of each of the source endoscope and the destination endoscope.

[0170] Fig. 19 shows an example of conversion when an endoscope C is added to the conversion between endoscope A and endoscope B shown in Fig. 15 etc. As shown in Fig. 19, the addition of endoscope C requires the preparation of new conversion formulas corresponding to each of the routes, route 31 (R31) from endoscope A to endoscope C and route 32 (R32) from endoscope B to endoscope C. In this way, the more endoscope models there are, the greater the number of conversion combinations becomes, and accordingly the corresponding man-hours required to prepare new conversion formulas become enormous.

[0171] (When there are three or more endoscopes) Image quality conversion of this embodiment when there are three or more endoscopes will be described with reference to Fig. 20. Fig. 19 is a diagram showing an example of image quality conversion of this embodiment when there are three endoscopes. As explained with reference to Fig. 19, if image quality conversion is performed based on direct conversion from the source to the destination, it will be necessary to add conversion formulas for all endoscopes as the number of models increases.

[0172] In the image quality converter 20 of this embodiment, the image quality of the source endoscope is first converted to intermediate image quality or standard image quality. Then, the image quality of the destination endoscope is converted from the intermediate image quality or standard image quality. As a result, the conversion formula for each endoscope only needs to correspond to the image quality of the endoscope and intermediate image quality, or the image quality of the endoscope and standard image quality. As a result, even if an endoscope is added, it is only necessary to add a conversion formula between the added endoscope and intermediate image quality, or between the added endoscope and standard image quality.

[0173] In the example shown in Fig. 20 , it is assumed that an endoscope C is added to endoscopes A and B. The conversion formulas corresponding to route 41 (R41) and route 42 (R42) have been prepared before endoscope C is added. In the example shown in Fig. 20 , when endoscope C is added, it is only necessary to prepare a new conversion formula corresponding to route 43 (R43). In other words, this eliminates the need for any adjustments on the endoscope A side and endoscope B side.

[0174] (Intermediate image quality and standard image quality) Intermediate image quality and standard image quality will now be explained. Intermediate image quality refers to the image quality of a representative endoscope, or the average or median of multiple representative endoscopes, or image quality generated according to a ratio arbitrarily set for each endoscope. For example, by using the average value of multiple representative endoscopes as the intermediate image quality, it is possible to reduce the amount of correction from the intermediate image quality to each endoscope.

[0175] Standard image quality is defined as the image quality of the raw data from an endoscope's image sensor, or an image quality close to that. For example, the image quality obtained from the output of a 4K RGB sensor or a 4K RGBW sensor (a sensor capable of simultaneous visible and infrared exposure) is considered standard image quality. No unnecessary signal processing is applied to raw data. Therefore, linear performance is guaranteed for raw data. As a result, image quality can be easily converted between endoscope models.

[0176] Intermediate quality images are also restored to a linear signal in the same way as standard quality images. However, the restoration may not be perfect. As a result, a certain amount of conversion error may be introduced into the intermediate quality images. As a result, when using intermediate quality images, adjustments may be required while checking the conversion results.

[0177] (Image display) Images displayed by a surgical image processing device equipped with an image quality converter are either images with the original image quality or images after conversion to the target image quality. In either case, when changing endoscopes, observations must take into account differences in the model, as the color tone and contrast tendencies vary depending on the endoscope.

[0178] Furthermore, when displaying the results of AI analysis, the displayed colors may be difficult to see depending on the model of endoscope and settings such as white balance and color adjustment. Although a method for selecting colors is sometimes provided, even in such cases, if different colors are selected for each endoscope, it may become difficult to determine which structure the displayed color refers to.

[0179] If the image quality converter has the function of converting the image quality to the default image quality of the endoscope, it converts the image quality to the default image quality of the specified endoscope and displays an image generated from the converted image. This allows observation to always be performed using images with familiar image quality. In addition, the display color of the AI ​​analysis results can always be the same and can be selected as an easy-to-understand color. The display color of the AI ​​analysis results can also be automatically set according to the set intermediate image quality or standard image quality. It may be specified to set an easy-to-see display color, or it may be specified not to allow selection of an difficult-to-see display color.

[0180] If the image quality converter has the function of converting image quality to intermediate image quality, an image generated from the intermediate image quality image is displayed. Generally, intermediate image quality is based on a neutral image quality with little emphasis, assuming conversion to various models. This allows stable observation in any scene. In addition, when displaying intermediate image quality, a corrected image may be displayed according to the user's preferences. This allows the same color to be selected for displaying the AI ​​analysis results, making the display easier to understand. The above content also applies when using standard image quality instead of intermediate image quality.

[0181] To make the processing in the image quality converter visible, an intermediate image quality image may be displayed, for example, on a monitor. For example, the intermediate image quality may be converted into an image of a preferred or familiar quality and displayed. The inferred drawing color may be difficult to see depending on the image quality of the endoscope. Therefore, by displaying an intermediate image quality image, the selected inferred drawing color can be easily seen regardless of the endoscope.

[0182] 1 Surgical image processing device 10 Preprocessing unit 12 Inference unit 14 Drawing unit 16 Display unit 20 Image quality converter 31 Instruction unit 32 Learning unit 100 Surgery support system 102 Camera (source of conversion) (endoscopic camera) 104 Display device 110 Output image 112 Converted image 120 Endoscopically captured image 121 Patch 122 Frame 124 Gradation Chart 125 Patch 127 Intersection 130 Graph 131 Gamma curve 150 White balance confirmation jig 151 Jig body 152 One bottom surface 154 Other bottom surface 156 Endoscope insertion port 158 ​​Cross section 160 Achromatic color chart 162 Skin color confirmation chart 164 Endoscope insertion port 166 Housing 168 Tip chart unit 200 Image before conversion 202 Image after conversion 204 Dark area 206 Light area

Claims

1. A surgical image processing device included in a surgical support system, comprising: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera; an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image; and a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image, wherein the image quality converter comprises: inverse gamma processing means that converts the output image into a linear signal; means for performing image quality conversion on the linear signal including at least one of color conversion, brightness conversion, noise reduction, and edge enhancement; and gamma processing means capable of performing gamma processing on the converted image, the gamma processing means being capable of performing or not performing the gamma processing.

2. The surgical image processing device according to claim 1, wherein the gamma processing means does not execute the gamma processing.

3. The surgical image processing device according to claim 1 or 2, wherein the image quality converter further comprises means for performing at least one of an offset adjustment, a brightness adjustment, and a white balance adjustment on the linear signal.

4. The surgical image processing device of claim 1, wherein the output image includes a non-linear signal that has been subjected to gamma processing and a linear signal that has not been subjected to gamma processing, and the image quality converter converts the image quality of the linear signal.

5. The surgical image processing device of claim 1, wherein the output image includes only a linear signal that has not been subjected to gamma processing, and the image quality converter converts the image quality of the linear signal.

6. The surgical image processing device according to claim 1, wherein the image quality converter further comprises means for performing image quality conversion including at least one of color conversion, luminance conversion, noise reduction, and edge enhancement on the nonlinear signal before converting the output image into a linear signal.

7. The surgical image processing device of claim 1, wherein the inference model is trained on an image in which the gamma processing has been applied to the converted image.

8. A surgical image processing device as claimed in claim 1 or 2, wherein the inverse gamma processing means inputs values ​​obtained by multiplying the reflectance or transmittance at multiple points in a gradation chart, in which multiple densities are defined, by different brightness ratios when the surgical images are captured, and outputs a graph showing the signal levels of the gradations at multiple points in the multiple surgical images captured at different brightness ratios, and the conditions for inverse gamma processing are determined from a graph normalized by setting the input at the point where the output is saturated to 1.

9. A surgical image processing device included in a surgical support system, comprising: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera; an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image; and a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image, wherein the camera is an endoscopic camera, and the image quality converter comprises: a white balance confirmation jig that utilizes the light source of the endoscopic camera; white balance error generation means that generates a white balance error based on instructions from a user or by automatically determining from the endoscopic camera image that the white balance confirmation jig is attached; and means for correcting a white balance correction value with the generated white balance error.

10. The surgical image processing device described in claim 9, wherein the inference model is trained using images in which the white balance correction value has been corrected based on the white balance error.

11. The surgical image processing device of claim 9, wherein the camera is an endoscopic camera, and the image quality converter comprises: a low saturation determination means for determining low saturation pixels, which are pixels of chromatic colors with low saturation in the output image, or low saturation regions, which are regions of chromatic colors with low saturation in the output image; a white balance gain generation means for generating a white balance gain for the low saturation pixels or a white balance gain for the low saturation region; a white balance error generation means for generating a white balance error for the low saturation pixels or a white balance error for a region in accordance with color temperature; and a means for correcting a white balance correction value with the generated white balance error.

12. The surgical image processing device described in claim 11, wherein the inference model is trained using images in which the white balance correction value has been corrected by the white balance error according to color temperature.

13. The surgical image processing device according to claim 9, wherein the camera is an endoscopic camera, and the image quality converter comprises: a skin color reproduction confirmation jig that is attached to the tip of the endoscopic camera, is shielded from light, and has skin colors of different brightness, hue angle, and saturation in multiple tones arranged in the center of the angle of view; a skin color reproduction error generating means that measures a skin color reproduction error based on an instruction from a user or by automatically determining that the skin color reproduction confirmation jig is attached; and a means for correcting at least one of a white balance correction value and a color conversion process using the generated skin color reproduction error.

14. The surgical image processing device described in claim 13, wherein the inference model is trained using images in which at least one of the white balance correction value and color conversion processing has been corrected based on the skin color reproduction error.

15. A surgical image processing device included in a surgical support system, comprising: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera; an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image; and a drawing unit that draws an image in which the analysis result by the inference model is reflected in the surgical image, wherein the image quality converter comprises: means for calculating an input level at which the signal level after image quality conversion of the output image is within a predetermined level range; means for setting an acceptance rate for pixels outside the predetermined level range; means for calculating the proportion of pixels outside the predetermined level range of the input image; means for comparing this proportion with the acceptance rate; and exposure correction means for correcting the exposure correction ratio in accordance with the proportion by which the acceptance rate is exceeded if the proportion exceeds the acceptance rate.

16. The surgical image processing device of claim 15, wherein the inference model is trained on images in which the exposure correction ratio has been corrected.

17. A surgical image processing device included in a surgical support system, comprising: a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera; an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image; and a drawing unit that draws an image in which the analysis results by the inference model are reflected in the surgical image, wherein the image quality converter comprises: a database that stores image quality characteristics for each camera; means for analyzing the characteristics of the output image from the camera; means for determining whether the analysis results match the image quality characteristics stored in the database; and means for automatically selecting the image quality conversion formula. The surgical image processing device described in claim 1, wherein the image quality converter comprises: a database that stores image quality characteristics for each camera; means for analyzing the characteristics of the output image from the camera; means for determining whether the analysis results match the image quality characteristics stored in the database; and means for automatically selecting the image quality conversion formula.

18. The surgical image processing device according to claim 17, wherein the inference model is trained on images whose image quality has been converted using the selected transformation formula.

19. A program that causes a surgical image processing device included in a surgical support system to perform the functions of a pre-processing unit including an image quality converter that converts the image quality of an output image of a surgical image acquired by a camera, an inference unit including an inference model that analyzes the converted image when the image after image quality conversion is treated as a converted image, and a drawing unit that draws an image in which the analysis results by the inference model are reflected in the surgical image, and causes the image quality converter to perform the function of converting the image quality of the output image of the surgical image acquired by the camera.

20. A system comprising an image quality converter that imports an image or video and converts the image quality, and an instruction unit that sets the source and destination of the image quality conversion.

21. The system according to claim 20, further comprising a learning unit that learns the image or video converted by the image quality converter.

22. The system according to claim 21, wherein the learning unit learns by mixing a plurality of the images or videos that have been converted in at least one of the source and destination in the image quality converter, the images or videos being converted differently.

23. The system described in claim 22, wherein the image or video is a surgical image captured by a camera, and the learning unit learns using the image quality converted image or video and correct answer data for the image or video as teacher data.

24. An image quality converter that converts the image quality of a surgical image acquired by an endoscope, the image quality converter having a function of converting the image quality of an image of a source endoscope to a default image quality of the source endoscope, and a function of converting the image quality of an image of a destination endoscope from the default image quality to a specific image quality of the destination endoscope, wherein the conversion of image quality between the source endoscope and the destination endoscope is performed by converting from the default image quality of the source endoscope to the default image quality of the destination endoscope.

25. An image quality converter that converts the image quality of a surgical image acquired by an endoscope, said image quality converter having a function of converting the image quality of an image of a source endoscope to an intermediate image quality, and a function of converting said intermediate image quality to the image quality of a destination endoscope, wherein the conversion of image quality between the source endoscope and the destination endoscope is performed via conversion via the intermediate image quality.

26. An image quality converter that converts the image quality of a surgical image acquired by an endoscope, said image quality converter having a function of converting the image quality of an image of a source endoscope to standard image quality, and a function of converting said standard image quality to the image quality of a destination endoscope, wherein the conversion of image quality between the source endoscope and the destination endoscope is performed via the standard image quality.

27. A surgical image processing device comprising: the image quality converter according to claim 24; and a function for displaying an image in the default image quality of the source endoscope, or an image in the default image quality of the destination endoscope.

28. A surgical image processing device comprising: the image quality converter according to claim 25; a function for displaying an image at the intermediate image quality; and a function for correcting the image quality of the displayed intermediate image quality to a desired image quality.

29. A surgical image processing device comprising: the image quality converter according to claim 26; a function for displaying an image in standard image quality; and a function for correcting the image quality of the displayed standard image quality to a desired image quality.

Images