Image processing method, device and equipment for 3D endoscope

Abstract

The invention provides an image processing method, device and equipment for a 3D endoscope, and the method comprises the steps: determining the first average brightness of a first current frame image and the second average brightness of a second current frame image, and enabling the first average brightness to be determined based on the brightness value of each pixel point of the first current frame image; determining a target change multiple based on the first average brightness and the target brightness, wherein the target change multiple represents an expected change proportion when the first average brightness is changed to the target brightness; determining a first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image, wherein the first exposure parameter comprises at least one of light source intensity, sensor gain and first algorithm digital gain; a second exposure parameter is determined based on the first average brightness, the second average brightness, and the first algorithm digital gain, the second exposure parameter including a second algorithm digital gain. According to the scheme, the internal space of the endoscope can be saved, and the three-dimensional white light image and the three-dimensional fluorescence image are supported at the same time.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and equipment for a 3D endoscope. Background Art

[0002] Endoscopes are a commonly used medical device, which consists of a light guide structure and a set of lenses. After the endoscope enters the interior of the target object, the endoscope can be used to collect the original image of the interior of the target object, and the target object can be inspected and treated based on the original image of the interior of the target object.

[0003] In order to support both three-dimensional white light images and three-dimensional fluorescence images, that is, to simultaneously collect three-dimensional white light images and three-dimensional fluorescence images inside the target object through the endoscope, it is necessary to deploy four image sensors in the endoscope, image sensor 1 collects white light image 1, image sensor 2 collects white light image 2, image sensor 3 collects fluorescence image 1, and image sensor 4 collects fluorescence image 2. White light image 1 and white light image 2 are fused to obtain a three-dimensional white light image, and fluorescence image 1 and fluorescence image 2 are fused to obtain a three-dimensional fluorescence image. However, due to the limited internal space of the endoscope, it may not be possible to deploy four image sensors inside the endoscope at the same time, which means that the endoscope cannot support three-dimensional white light images and three-dimensional fluorescence images. Summary of the invention

[0004] The present application provides an image processing method for a 3D endoscope. If the 3D endoscope adopts a white fluorescent stroboscopic mode, white light and fluorescence are alternately output by a light source; when the light source outputs white light, a first white light image is collected by a first sensor, and a second white light image is collected by a second sensor; when the light source outputs fluorescence, a first fluorescence image is collected by the first sensor, and a second fluorescence image is collected by the second sensor. The method includes: acquiring a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image; determining a first average brightness of the first current frame image and a second average brightness of the second current frame image, wherein the first average brightness is determined based on the brightness value of each pixel point of the first current frame image Determine; determine a target change multiple based on the first average brightness and the configured target brightness, the target change multiple represents the expected change ratio when changing from the first average brightness to the target brightness; determine a first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image, the first exposure parameter includes at least one of light source intensity, sensor gain and first algorithm digital gain; determine a second exposure parameter based on the first average brightness, the second average brightness and the first algorithm digital gain, the second exposure parameter includes a second algorithm digital gain; wherein, if the first current frame image is a first white light image, the first exposure parameter and the second exposure parameter are exposure parameters of a next frame of white light image; if the first current frame image is a first fluorescent image, the first exposure parameter and the second exposure parameter are exposure parameters of a next frame of fluorescent image.

[0005] The present application provides an image processing device for a 3D endoscope. If the 3D endoscope adopts a white fluorescent stroboscopic mode, white light and fluorescence are alternately output by a light source; when the light source outputs white light, a first white light image is collected by a first sensor, and a second white light image is collected by a second sensor; when the light source outputs fluorescence, a first fluorescence image is collected by the first sensor, and a second fluorescence image is collected by the second sensor. The device includes: an acquisition module, used to acquire a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image; a determination module, used to determine a first average brightness of the first current frame image and a second average brightness of the second current frame image, wherein the first average brightness is based on each pixel point of the first current frame image a brightness value determined by the first average brightness and the configured target brightness; a target change multiple is determined based on the first average brightness and the configured target brightness, the target change multiple represents the expected change ratio when the first average brightness changes to the target brightness; a control module is used to determine a first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image, the first exposure parameter includes at least one of the light source intensity, the sensor gain and the first algorithm digital gain; a second exposure parameter is determined based on the first average brightness, the second average brightness and the first algorithm digital gain, the second exposure parameter includes the second algorithm digital gain; wherein, if the first current frame image is a first white light image, the first exposure parameter and the second exposure parameter are the exposure parameters of the next white light image; if the first current frame image is a first fluorescent image, the first exposure parameter and the second exposure parameter are the exposure parameters of the next fluorescent image.

[0006] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method of the 3D endoscope of the above example of the present application.

[0007] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image processing method of the 3D endoscope of the above example of the present application is implemented.

[0008] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the image processing method of the 3D endoscope of the above example of the present application.

[0009] It can be seen from the above technical solutions that in the embodiment of the present application, only two image sensors need to be deployed simultaneously inside the endoscope, and four image sensors do not need to be deployed simultaneously inside the endoscope, thereby saving the internal space of the endoscope. The light source alternately outputs white light and fluorescence. When the light source outputs white light, the first white light image is collected by the first sensor, and the second white light image is collected by the second sensor. The first white light image and the second white light image are fused to obtain a three-dimensional white light image. When the light source outputs fluorescence, the first fluorescence image is collected by the first sensor, and the second fluorescence image is collected by the second sensor. The first fluorescence image and the second fluorescence image are fused to obtain a three-dimensional fluorescence image, so that the endoscope supports three-dimensional white light images and three-dimensional fluorescence images almost simultaneously. The exposure parameters (such as light source intensity) of the next frame image (white light image or fluorescence image) can be controlled, thereby linking the light source of the endoscope, that is, controlling the light source intensity of the light source, ensuring the practicality and stability of the exposure function, and not prone to overexposure, thereby improving the use experience of medical staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of an image processing method for a 3D endoscope in one embodiment of the present application;

[0011] Figure 2A is a schematic structural diagram of an endoscope system in one embodiment of the present application;

[0012] Figure 2B is a functional structure diagram of an endoscope system in one embodiment of the present application;

[0013] Figure 3 is a schematic diagram of the overall framework of a multi-mode automatic exposure method in one embodiment of the present application;

[0014] Figure 4 It is a flowchart of the image processing method of the 3D endoscope in the white fluorescence stroboscopic mode in the present application;

[0015] Figure 5A is a schematic diagram of an endoscope imaging area in one embodiment of the present application;

[0016] Figure 5B is a schematic diagram of an endoscope imaging area in one embodiment of the present application;

[0017] Figure 6 It is a flowchart of an image processing method of a 3D endoscope in a white light mode according to an embodiment of the present application;

[0018] Figure 7 is a schematic structural diagram of an image processing device for a 3D endoscope in one embodiment of the present application;

[0019] Figure 8 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0020] In the embodiment of the present application, a 3D endoscope image processing method is proposed. If the 3D endoscope (in the subsequent process, the 3D endoscope is referred to as the endoscope) adopts the white fluorescence stroboscopic mode, the light source of the endoscope alternately outputs white light and fluorescence. When the light source outputs white light, the first white light image is collected by the first sensor of the endoscope, and the second white light image is collected by the second sensor of the endoscope. When the light source outputs fluorescence, the first fluorescence image is collected by the first sensor, and the second fluorescence image is collected by the second sensor.

[0021] See also Figure 1 FIG. 1 is a flow chart of the method, which may include:

[0022] Step 101, acquiring a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image.

[0023] Step 102: determine a first average brightness of the first current frame image and a second average brightness of the second current frame image, wherein the first average brightness is determined based on the brightness value of each pixel of the first current frame image.

[0024] Step 103: determine a target change multiple based on the first average brightness and the configured target brightness, where the target change multiple represents an expected change ratio when changing from the first average brightness to the target brightness.

[0025] Step 104: determine the first exposure parameter of the next frame image based on the target change multiple and the exposure parameter of the first current frame image, wherein the first exposure parameter includes at least one of the light source intensity, the sensor gain, and the first algorithm digital gain; determine the second exposure parameter of the next frame image based on the first average brightness, the second average brightness, and the first algorithm digital gain, wherein the second exposure parameter includes the second algorithm digital gain.

[0026] Among them, if the first current frame image is the first white light image, the first exposure parameter and the second exposure parameter may be the exposure parameters of the next white light image; if the first current frame image is the first fluorescent image, the first exposure parameter and the second exposure parameter may be the exposure parameters of the next fluorescent image.

[0027] Exemplarily, determining the first average brightness of the first current frame image may include, but is not limited to: dividing the first current frame image into a plurality of image blocks, and for each image block, determining the block average brightness of the image block based on the brightness value of each pixel of the image block. Obtain a first image block located in an effective field of view area and a second image block not located in the effective field of view area. For example, the effective field of view area may be an effective illumination area of ​​a light source. Determine the first average brightness based on the block average brightness of non-highlight blocks in the first image block, the block average brightness of highlighted blocks in the first image block, the acquired target overexposure weight, and the total number of first image blocks; the target overexposure weight may represent the weight corresponding to the block average brightness of the highlighted blocks.

[0028] Before acquiring the first image block located in the effective field of view area and the second image block not located in the effective field of view area, a brightness threshold may also be determined based on the central area brightness of the first current frame image and the configured brightness threshold ratio, where the central area brightness is the average brightness value of each image block in the central area. A first boundary and a second boundary of the first dimension are determined, and a third boundary and a fourth boundary of the second dimension are determined based on the first boundary, the second boundary, and the width and height of the first current frame image.

[0029] Among them, if the first dimension is the column dimension and the second dimension is the row dimension, the first boundary is the first column that meets the preset conditions from the left, and the second boundary is the first column that meets the preset conditions from the right; if the first dimension is the row dimension and the second dimension is the column dimension, the first boundary is the first row that meets the preset conditions from the top, and the second boundary is the first row that meets the preset conditions from the bottom; a column that meets the preset conditions means that the average value of the block average brightness of all image blocks in the column is not less than the brightness threshold; a row that meets the preset conditions means that the average value of the block average brightness of all image blocks in the row is not less than the brightness threshold.

[0030] If the image block is located in the area formed by the first boundary, the second boundary, the third boundary and the fourth boundary, the image block is the first image block; otherwise, the image block is the second image block.

[0031] Exemplarily, the method for determining the target overexposure weight may include, but is not limited to: determining the target highlight ratio based on the total number of highlight blocks and the total number of first image blocks; wherein, for each first image block, if the block average brightness of the first image block is not less than the highlight block brightness threshold, then the first image block belongs to the highlight block, and if the block average brightness of the first image block is less than the highlight block brightness threshold, then the first image block does not belong to the highlight block. Then, the target overexposure weight is determined based on the target highlight ratio; wherein, the larger the target highlight ratio, the larger the target overexposure weight; wherein, multiple threshold intervals may be preconfigured, and different threshold intervals correspond to different mapping relationships, and the mapping relationship represents the relationship between the highlight ratio and the overexposure weight; the target overexposure weight is obtained by querying the mapping relationship corresponding to the threshold interval to which the target highlight ratio belongs.

[0032] Exemplarily, determining the target change multiple based on the first average brightness and the configured target brightness may include, but is not limited to: correcting the first average brightness based on the brightness information of the historical frame image to obtain the brightness estimation value of the first current frame image. Determining the first difference between the target brightness and the brightness estimation value; determining the target change multiple of the first current frame image based on the first difference and the second difference; wherein the second difference may be the difference between the target brightness and the brightness estimation value of the historical frame image. For example, if the first current frame image is a first white light image, the historical frame image may be a historical frame white light image, and if the first current frame image is a first fluorescent image, the historical frame image may be a historical frame fluorescent image.

[0033] Exemplarily, the brightness information of the historical frame image may include the average brightness of the historical frame image and the brightness estimation value of the historical frame image. The first average brightness is corrected based on the brightness information of the historical frame image to obtain the brightness estimation value of the first current frame image. It may include but is not limited to: determining the brightness prediction value of the first current frame image based on the average brightness of the historical frame image, performing a weighted operation on the first average brightness and the brightness prediction value to obtain an average brightness correction value; performing a weighted operation on the brightness estimation value of the historical frame image and the average brightness correction value to obtain the brightness estimation value of the first current frame image.

[0034] Exemplarily, determining the target change multiple of the first current frame image based on the first difference and the second difference may include: determining the brightness control amount corresponding to the brightness estimation value based on the first difference, the second difference, the configured proportional adjustment coefficient and the configured differential adjustment coefficient; determining the initial change multiple of the first current frame image based on the brightness control amount and the brightness estimation value; performing a weighted operation on the initial change multiple of the first current frame image and the target change multiple of the historical frame images to obtain the target change multiple of the first current frame image.

[0035] Exemplarily, determining the first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image may include but is not limited to: if the target change multiple is greater than a preset change multiple threshold, increasing the light source intensity in the exposure parameter of the first current frame image to obtain the light source intensity in the first exposure parameter; and / or, increasing the sensor gain in the exposure parameter of the first current frame image to obtain the sensor gain in the first exposure parameter; and / or, increasing the first algorithm digital gain in the exposure parameter of the first current frame image to obtain the first algorithm digital gain in the first exposure parameter; if the target change multiple is less than the preset change multiple threshold, reducing the light source intensity in the exposure parameter of the first current frame image to obtain the light source intensity in the first exposure parameter; and / or, reducing the sensor gain in the exposure parameter of the first current frame image to obtain the sensor gain in the first exposure parameter; and / or, reducing the first algorithm digital gain in the exposure parameter of the first current frame image to obtain the first algorithm digital gain in the first exposure parameter.

[0036] Determining the second algorithm digital gain based on the first average brightness, the second average brightness and the first algorithm digital gain may include but is not limited to: determining a quotient between the second average brightness and the first average brightness; if the quotient is less than a preset ratio threshold, determining the second algorithm digital gain based on the product value between the first algorithm digital gain and the quotient; or, if the quotient is not less than the preset ratio threshold, determining the first algorithm digital gain as the second algorithm digital gain.

[0037] Exemplarily, if the endoscope adopts the white light mode, white light can be continuously output through the light source, a third white light image can be collected through the first sensor, and a fourth white light image can be collected through the second sensor, based on which, a third average brightness of the third white light image and a fourth average brightness of the fourth white light image can be determined; a target change multiple is determined based on the third average brightness and the target brightness; a third exposure parameter of the next frame of white light image is determined based on the target change multiple and the exposure parameter of the third white light image, and the third exposure parameter may include at least one of a shutter time, a sensor gain, and a third algorithm digital gain; a fourth exposure parameter of the next frame of white light image is determined based on the third average brightness, the fourth average brightness, and the third algorithm digital gain, and the fourth exposure parameter may include a fourth algorithm digital gain.

[0038] Exemplarily, if the endoscope switches from the white light mode to the white fluorescent stroboscopic mode, for the mode switching process, before acquiring the first current frame image and the second current frame image for the first time, the first exposure parameter is set to the first fixed exposure parameter, and the second exposure parameter is set to the second fixed exposure parameter. When the first current frame image and the second current frame image are acquired for the Mth time, the correction coefficient corresponding to M is determined, and M can be a positive integer; wherein the larger M is, the larger the correction coefficient is; after determining the first exposure parameter and the second exposure parameter, the updated first exposure parameter is determined based on the first exposure parameter and the correction coefficient; and the updated second exposure parameter is determined based on the second exposure parameter and the correction coefficient.

[0039] Exemplarily, if the endoscope switches from the white fluorescent stroboscopic mode to the white light mode, for the mode switching process, before acquiring the third white light image and the fourth white light image for the first time, the third exposure parameter can be set to the third fixed exposure parameter, and the fourth exposure parameter can be set to the fourth fixed exposure parameter. When the third white light image and the fourth white light image are acquired for the Nth time, the correction coefficient corresponding to N can be determined, and N can be a positive integer; wherein the larger N is, the larger the correction coefficient is; after determining the third exposure parameter and the fourth exposure parameter, the updated third exposure parameter is determined based on the third exposure parameter and the correction coefficient; and the updated fourth exposure parameter is determined based on the fourth exposure parameter and the correction coefficient.

[0040] It can be seen from the above technical solutions that in the embodiment of the present application, only two image sensors need to be deployed simultaneously inside the endoscope, and four image sensors do not need to be deployed simultaneously inside the endoscope, thereby saving the internal space of the endoscope. The light source alternately outputs white light and fluorescence. When the light source outputs white light, the first white light image is collected by the first sensor, and the second white light image is collected by the second sensor. The first white light image and the second white light image are fused to obtain a three-dimensional white light image. When the light source outputs fluorescence, the first fluorescence image is collected by the first sensor, and the second fluorescence image is collected by the second sensor. The first fluorescence image and the second fluorescence image are fused to obtain a three-dimensional fluorescence image, so that the endoscope supports both three-dimensional white light images and three-dimensional fluorescence images. The exposure parameters (such as light source intensity) of the next frame image (white light image or fluorescence image) can be controlled, thereby linking the light source of the endoscope, that is, controlling the light source intensity of the light source, ensuring the practicality and stability of the exposure function, and not prone to overexposure, thereby improving the use experience of medical staff.

[0041] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0042] Endoscopes are a commonly used medical device. Endoscopes can be composed of a light guide structure and a set of lenses. Endoscopes can be inserted into the target object (such as a patient waiting for a specimen) to collect images inside the target object and output the images to an external display device and storage device. Users (such as doctors, etc.) can observe the images displayed by the display device to check the bleeding site, tumor site, abnormal site, etc. inside the target object, and provide real-time images of surgical treatment. Users (such as doctors, etc.) can access the video in the storage device for postoperative review and surgical training.

[0043] See also Figure 2A The figure shows a schematic diagram of the general structure of the endoscope system. The endoscope system may include an endoscope, a light source, a camera system host, a display device, and a storage device. The endoscope is inserted into the interior of a target object to photograph the observation area inside the target object and generate image data. The light source provides illumination light emitted from the front end of the endoscope. The camera system host can perform image processing on the image data generated by the endoscope and uniformly control the overall operation of the endoscope system. The display device can display the image corresponding to the image data generated by the endoscope. The storage device can store the image corresponding to the image data generated by the endoscope.

[0044] See also Figure 2B The figure shows a functional structure diagram of an endoscope system. The endoscope has an imaging optical system, an imaging unit, a processing unit and an operating unit. The imaging optical system focuses light from the observation site, and the imaging optical system is composed of one or more lenses. The imaging unit generates image data by photoelectric conversion of the light received by each pixel. The imaging unit is composed of image sensors such as CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge-coupled Device). The processing unit converts the image data generated by the imaging unit into a digital signal, and sends the converted signal to the host of the imaging system. The operating unit receives an indication signal for switching the action of the endoscope, receives a signal for switching the light source to perform illumination light switching, and outputs the indication signal to the host of the imaging system. The operating unit may include but is not limited to a switch, a button and a touch panel.

[0045] The light source may include an illumination control unit and an illumination unit. The illumination control unit may receive an indication signal from the camera system host, and control the illumination unit to provide illumination light to the endoscope based on the indication signal. The camera system host processes the image data received from the endoscope and transmits it to a display device and a storage device. The display device and the storage device are external devices. The camera system host includes an image input unit, an image processing unit, an intelligent processing unit, a video encoding unit, a control unit, and an operation unit. The image input unit receives a signal sent by the endoscope and transmits the received signal to the image processing unit. The image processing unit performs an ISP (Image Signal Processor) operation on the image of the image input unit, and the ISP operation includes but is not limited to brightness conversion, sharpening, de-moiré, and scaling. The image processed by the image processing unit is transmitted to the intelligent processing unit, the video encoding unit, or the display device. The intelligent processing unit performs intelligent analysis on the image processed by the image processing unit, and the intelligent analysis includes but is not limited to scene classification based on deep learning, instrument head detection, gauze detection, moiré classification, and dense fog classification. The image processed by the intelligent processing unit is transmitted to the image processing unit or the video encoding unit.

[0046] The image processing unit processes the image processed by the intelligent processing unit, and the processing methods include but are not limited to brightness transformation, moiré removal, frame stacking and scaling. The video encoding unit encodes and compresses the image processed by the image processing unit or the intelligent processing unit, and transmits the encoded and compressed image to the storage device. The control unit controls the various modules of the endoscope system, including but not limited to the lighting method of the light source, the image processing method, the intelligent processing method and the video encoding method. The input unit includes but is not limited to switches, buttons and touch panels, accepts external indication signals, and outputs the accepted indication signals to the control unit.

[0047] In a possible implementation, the endoscope may support two-dimensional white light images. For example, an image sensor may be deployed inside the endoscope, and a light source may continuously emit white light (white light may also be called visible light), and the image sensor may collect a two-dimensional white light image inside the target object.

[0048] In a possible implementation, the endoscope can support three-dimensional white light images. For example, two image sensors can be deployed simultaneously inside the endoscope, white light is continuously emitted by a light source, a two-dimensional white light image 1 of the inside of the target object is collected by image sensor 1, and a two-dimensional white light image 2 of the inside of the target object is collected by image sensor 2, and the two-dimensional white light image 1 and the two-dimensional white light image 2 are fused to obtain a three-dimensional white light image, and there is no restriction on the fusion method of the two-dimensional white light image 1 and the two-dimensional white light image 2.

[0049] In a possible implementation, an endoscope (also called a fluorescence endoscope) can support two-dimensional white light images and two-dimensional fluorescence images at the same time. For example, two image sensors are deployed inside the endoscope at the same time (the two image sensors are used to respond to white light and fluorescence respectively, and fluorescence can be excited by infrared light).

[0050] The light source needs to emit white light and fluorescence at the same time. When the light hits the object, the reflected light enters the camera optical system. The camera optical system refracts white light and fluorescence of different wavelengths differently, so that the reflected white light hits the image sensor 1 and the reflected fluorescence hits the image sensor 2. In this way, the two-dimensional white light image inside the target object is collected by the image sensor 1. The two-dimensional fluorescence image inside the target object is collected by the image sensor 2. On this basis, the two-dimensional white light image and the two-dimensional fluorescence image can also be fused. For example, the two-dimensional white light image is dyed based on the two-dimensional fluorescence image, and the two-dimensional fluorescence image is displayed on the two-dimensional white light image. There is no restriction on this fusion method.

[0051] In a possible implementation, an endoscope (also referred to as a fluorescence endoscope) supports both three-dimensional white light images and three-dimensional fluorescence images. For example, four image sensors can be deployed simultaneously inside the endoscope. The light source needs to emit white light and fluorescence at the same time. The light hits the object, and the reflected light enters the camera optical system. The camera optical system refracts white light and fluorescence of different wavelengths differently, so that the reflected white light hits image sensor 1 and image sensor 2, and the reflected fluorescence hits image sensor 3 and image sensor 4. In this way, a two-dimensional white light image 1 inside the target object can be collected by image sensor 1, and a two-dimensional white light image 2 inside the target object can be collected by image sensor 2. The two-dimensional white light image 1 and the two-dimensional white light image 2 are fused to obtain a three-dimensional white light image. In addition, a two-dimensional fluorescence image 1 inside the target object can be collected by image sensor 3, and a two-dimensional fluorescence image 2 inside the target object can be collected by image sensor 4. The two-dimensional fluorescence image 1 and the two-dimensional fluorescence image 2 are fused to obtain a three-dimensional fluorescence image. On this basis, the three-dimensional white light image and the three-dimensional fluorescence image can also be fused. For example, the three-dimensional white light image can be dyed based on the three-dimensional fluorescence image, and the three-dimensional fluorescence image can be displayed on the three-dimensional white light image. There is no restriction on this fusion method.

[0052] In summary, in order to simultaneously support 3D white light images and 3D fluorescence images, four image sensors need to be deployed in the endoscope. However, due to the limited internal space of the endoscope, it is impossible to simultaneously deploy four image sensors inside the endoscope, resulting in the endoscope being unable to support 3D white light images and 3D fluorescence images.

[0053] In view of the above findings, in this embodiment, two image sensors are deployed simultaneously inside the endoscope, and the endoscope in which two image sensors are deployed simultaneously is called a 3D endoscope, and is hereinafter referred to as an endoscope. If the endoscope adopts a white fluorescent stroboscopic mode, three-dimensional white light images and three-dimensional fluorescent images are supported by two image sensors. If the endoscope adopts a white light mode, three-dimensional white light images can be supported by two image sensors.

[0054] Exemplarily, if the endoscope adopts a white fluorescence stroboscopic mode (called a fusion mode), the light source alternately outputs white light and fluorescence (it should be noted that the light source does not emit white light and fluorescence at the same time, but first outputs white light, then outputs fluorescence, then outputs white light, then outputs fluorescence, and so on). When the light source outputs white light, the white light hits the object, and the reflected light enters the camera optical system. The camera optical system refracts white light of different wavelengths differently, so that the reflected white light hits image sensor 1 and image sensor 2. A two-dimensional white light image 1 (hereinafter referred to as a first white light image) of the inside of the target object is collected by image sensor 1, and a two-dimensional white light image 2 (hereinafter referred to as a second white light image) of the inside of the target object is collected by image sensor 2. The first white light image and the second white light image are fused to obtain a three-dimensional white light image. Then, when the light source outputs fluorescence, the fluorescence hits the object, and the reflected light enters the camera optical system. The camera optical system refracts fluorescence of different wavelengths differently, so that the reflected fluorescence hits image sensor 1 and image sensor 2. A two-dimensional fluorescence image 1 (hereinafter referred to as the first fluorescence image) inside the target object is collected through image sensor 1, and a two-dimensional fluorescence image 2 (hereinafter referred to as the second fluorescence image) inside the target object is collected through image sensor 2. The first fluorescence image and the second fluorescence image are fused to obtain a three-dimensional fluorescence image.

[0055] Although the 3D white light image and the 3D fluorescence image do not correspond to the same moment (e.g., the time difference between the 3D white light image and the 3D fluorescence image of the previous and next two frames is about 16ms), however, considering that the movement speed of the endoscope is usually slow and the movement amplitude is usually small in medical application scenarios, the 3D white light image and the 3D fluorescence image can be equivalent to the white light image and the fluorescence image at the same moment, so that the adjacent 3D white light image and the 3D fluorescence image can be fused, for example, the 3D white light image can be stained based on the 3D fluorescence image. For example, the 3D white light image at time t1 and the 3D fluorescence image at time t2 can be fused, the 3D white light image at time t3 and the 3D fluorescence image at time t4 can be fused, and so on.

[0056] Exemplarily, if the endoscope adopts the white light mode, the light source continuously outputs white light (i.e., only outputs white light), the white light hits the object, and the reflected light enters the camera optical system. The camera optical system refracts white light of different wavelengths differently, so that the reflected white light hits the image sensor 1 and the image sensor 2. The two-dimensional white light image 3 (hereinafter referred to as the third white light image) inside the target object is collected through the image sensor 1, and the two-dimensional white light image 4 (hereinafter referred to as the fourth white light image) inside the target object is collected through the image sensor 2. The third white light image and the fourth white light image are fused to obtain a three-dimensional white light image.

[0057] In summary, only two image sensors need to be deployed in the endoscope to support the white fluorescence stroboscopic mode and the white light mode. In the white fluorescence stroboscopic mode, both three-dimensional white light images and three-dimensional fluorescence images are supported. In the white light mode, three-dimensional white light images are supported. For the white fluorescence stroboscopic mode, the light source is a stroboscopic light source that flashes white light and fluorescence sequentially. The image sensor responds to white light and fluorescence alternately in time, and then fuses the previous and next two frames (one frame of three-dimensional white light image and one frame of three-dimensional fluorescence image). If the white fluorescence stroboscopic mode is not linked to the light source, overexposure is prone to occur, which greatly affects the doctor's visual perception and endangers the life safety of the target object to a certain extent. Therefore, an exposure method of a linked light source is adopted. For the white light mode, the light source is a constantly bright white light source, and the image sensor only responds to white light to obtain a three-dimensional white light image. The white light mode does not require a linked light source.

[0058] In the embodiment of the present application, a multi-mode automatic exposure method of an endoscope with a linked light source is proposed. Figure 3 As shown, it is a schematic diagram of the overall framework. The imaging unit can be composed of one or more lenses, optical fibers, two image sensors (hereinafter referred to as the first image sensor and the second image sensor) and a metal structure. The lens is used to filter the incident light, the optical fiber is used to guide the light, the metal structure is used to protect the lens and the optical fiber, and the two image sensors are used to sense light and respond to white light and fluorescence. The image processing unit is used to implement the image processing method of the 3D endoscope (the image processing method is shown in the subsequent embodiments). The light source unit, that is, the light source itself, can continuously output white light, or can alternately output white light and fluorescence. The visible object surface can be the visible object surface inside the target object, or it can be the visible object surface in other scenes, without limitation.

[0059] In the above application scenarios, an image processing method for a 3D endoscope is proposed in an embodiment of the present application, which may involve an image processing method in a white fluorescent stroboscopic mode, an image processing method in a white light mode, and an image processing method in a mode switching process. For example, the endoscope supports two exposure modes, namely a white fluorescent stroboscopic mode and a white light mode, and the user can switch between the two exposure modes at will. For example, if the user triggers the use of the white fluorescent stroboscopic mode, the image processing method in the white fluorescent stroboscopic mode can be executed. If the user triggers the use of the white light mode, the image processing method in the white light mode can be executed. In the process of switching between the two exposure modes, the image processing method in the mode switching process can be executed.

[0060] Case 1: Image processing method in white fluorescent stroboscopic mode.

[0061] See also Figure 4 FIG. 1 is a flow chart of an image processing method in a white fluorescent stroboscopic mode. The method can be applied to a camera system host (such as an image processing unit). The process may include:

[0062] Step 401: Acquire a first current frame image and a second current frame image.

[0063] For example, in the white fluorescent stroboscopic mode, the light source is switched to a stroboscopic light source, that is, the light source (stroboscopic light source) alternately outputs white light and fluorescence, for example, alternately outputs white light and fluorescence at the same interval in a time-division manner. The image sensor senses the environment at the same frequency as the stroboscopic light source, obtains a white light image or a fluorescent image, and sends the white light image or the fluorescent image to the camera system host.

[0064] For example, when the light source outputs white light, the first white light image is collected by the first sensor, and the second white light image is collected by the second sensor. Based on this, when the host of the camera system acquires the first current frame image and the second current frame image, the first current frame image is the first white light image, and the second current frame image is the second white light image. The first sensor can be a left-eye sensor, and the second sensor can be a right-eye sensor, or the first sensor can be a right-eye sensor, and the second sensor can be a left-eye sensor. When the light source outputs fluorescence, the first fluorescent image is collected by the first sensor, and the second fluorescent image is collected by the second sensor. Based on this, when the host of the camera system acquires the first current frame image and the second current frame image, the first current frame image is the first fluorescent image, and the second current frame image is the second fluorescent image.

[0065] Step 402, image brightness statistics. In the image brightness statistics process, a first average brightness of the first current frame image can be determined, and the first average brightness can be determined based on the brightness value of each pixel of the first current frame image. In addition, a second average brightness of the second current frame image can be determined, and the second average brightness can be determined based on the brightness value of each pixel of the second current frame image.

[0066] In a possible implementation, for an image brightness statistics process, the process may include:

[0067] Step S11: Divide the first current frame image into a plurality of image blocks, and for each image block, determine the block average brightness of the image block based on the brightness value of each pixel of the image block.

[0068] For example, the first current frame image can be divided into m*n image blocks, where m means that there are m image blocks in the horizontal direction of the first current frame image, and n means that there are n image blocks in the vertical direction of the first current frame image, where m is a positive integer greater than 1, and n is a positive integer greater than 1. The size of each image block can be the same, such as each image block is an image block of 4*4 pixels. m and n can be the same, or they can be different.

[0069] For each image block, the brightness value of each pixel of the image block may be determined, and the average brightness value of all pixels of the image block is taken as the block average brightness of the image block.

[0070] Step S12: determining a brightness threshold based on a ratio of the brightness of a central area of ​​the first current frame image to a configured brightness threshold, wherein the brightness of the central area is an average value of brightness values ​​of each image block in the central area.

[0071] For example, taking the central image block of the first current frame image as the center, select an area of ​​a specified size as the central area. For example, select a circular area with a radius of a as the central area, and for another example, select a rectangular area with a width of b1 and a height of b2 as the central area, and b1 and b2 can be the same or different. For example, the units of radius a, width b1, and height b2 all represent the number of image blocks. After obtaining the central area, the brightness value of each image block in the central area can be determined, and the average brightness value of all image blocks in the central area can be used as the brightness of the central area.

[0072] For example, the brightness threshold ratio may be preconfigured, the brightness threshold ratio may be a value greater than 1, and the brightness threshold ratio may be configured based on experience, and there is no restriction on the brightness threshold ratio.

[0073] The brightness threshold may be determined based on the central area brightness and the brightness threshold ratio, for example, the quotient of the central area brightness and the brightness threshold ratio is used as the brightness threshold. For example, the central area brightness is recorded as y_ave, and the brightness threshold ratio is recorded as y_ratio, then the brightness threshold is y_ave / y_ratio.

[0074] Step S13, determining a first boundary and a second boundary of the first dimension, and determining a third boundary and a fourth boundary of the second dimension based on the first boundary, the second boundary, and the width and height of the first current frame image.

[0075] For example, the imaging area of ​​the endoscope may be a circular area or a partial circular area, see Figure 5A The figure shows a schematic diagram of the endoscope imaging area. The circular area is the actual visible scene, and the background area is a black area without light, and there is often light leakage in the background area. In order to better calculate the brightness of the first current frame image, it is necessary to exclude the interference of the background area. Therefore, the effective field of view area (i.e., the field of view circle, which is the effective illumination area of ​​the light source) can be detected, and the brightness of the image block in the effective field of view area can be calculated. Figure 5B Shown is a schematic diagram of the endoscope imaging area.

[0076] In order to detect the effective viewing area, it is necessary to determine the first boundary, the second boundary, the third boundary and the fourth boundary of the effective viewing area, and the inner area of ​​these boundaries is the effective viewing area.

[0077] In a possible implementation, if the first dimension is a column dimension and the second dimension is a row dimension, then the first boundary is the first column from the left that meets the preset condition, and the second boundary is the first column from the right that meets the preset condition. For example, starting from the left, traversing the image blocks in sequence, for the currently traversed column, if the average value of the block average brightness of all image blocks in the column is less than the brightness threshold, then the column is not the first boundary. If the average value of the block average brightness of all image blocks in the column is not less than the brightness threshold, then the column is a column that meets the preset condition, that is, the column is the first boundary. Similarly, starting from the right, traversing in sequence until the average value of the block average brightness of all image blocks in the column is not less than the brightness threshold, that is, the column is the second boundary.

[0078] After obtaining the first boundary and the second boundary, the third boundary of the row dimension can be determined based on the first boundary, the width and the height of the first current frame image. For example, the quotient of the first boundary (indicating the nth image block in the horizontal direction) and the width is equal to the quotient of the third boundary (indicating the nth image block in the vertical direction) and the height.

[0079] Furthermore, the fourth boundary of the row dimension can be determined based on the second boundary, the width and the height of the first current frame image. For example, the quotient of the second boundary and the width is equal to the quotient of the fourth boundary and the height.

[0080] In summary, it can be seen that in the process of effective field of view detection, the average value y of the block average brightness of all image blocks in each column is calculated column by column. i (i=1,…,n), then y i Compared with the brightness threshold (y_ave / y_ratio), if y i If the brightness is greater than or equal to the brightness threshold, it is recorded as a valid image block in the effective field of view. In this way, the left and right boundaries of the effective field of view can be obtained, and the upper and lower boundaries of the effective field of view are calculated based on the left and right boundaries and the image aspect ratio w / h (that is, the ratio between the width w and the height h).

[0081] In a possible implementation, if the first dimension is a row dimension and the second dimension is a column dimension, then the first boundary is the first row that meets the preset condition starting from the top, and the second boundary is the first row that meets the preset condition starting from the bottom. For example, starting from the top, traversing the image blocks in sequence, for the currently traversed row, if the average value of the block average brightness of all image blocks in the row is less than the brightness threshold, then the row is not the first boundary. If the average value of the block average brightness of all image blocks in the row is not less than the brightness threshold, then the row is a row that meets the preset condition, that is, the row is the first boundary. Similarly, starting from the bottom, traversing in sequence until the average value of the block average brightness of all image blocks in the row is not less than the brightness threshold, that is, the row is the second boundary.

[0082] After the first boundary and the second boundary are obtained, the third boundary and the fourth boundary in the row dimension may be determined based on the first boundary, the second boundary, and the width and height of the first current frame image.

[0083] Step S14, obtaining a first image block located in the effective field of view area and a second image block not located in the effective field of view area. For example, for each image block, if the image block is located in the effective field of view area formed by the first boundary, the second boundary, the third boundary and the fourth boundary, then the image block is the first image block (i.e., the effective image block); otherwise, if the image block is not located in the effective field of view area formed by the first boundary, the second boundary, the third boundary and the fourth boundary, then the image block is the second image block (i.e., the invalid image block).

[0084] Step S15: determining the target highlight ratio based on the total number of highlight blocks and the total number of first image blocks.

[0085] Exemplarily, a highlight block brightness threshold (denoted as high_light_thr) may be pre-configured, and no restriction is imposed on the highlight block brightness threshold. For each first image block, if the block average brightness of the first image block is not less than the highlight block brightness threshold, the first image block belongs to a highlight block; if the block average brightness of the first image block is less than the highlight block brightness threshold, the first image block does not belong to a highlight block. After performing the above processing on all first image blocks, the total number of highlight blocks (denoted as high_light_blk_num) may be counted. Obviously, the total number of highlight blocks is less than or equal to the total number of first image blocks.

[0086] The quotient of the total number of highlight blocks and the total number of first image blocks may be used as the target highlight ratio. For example, the target highlight ratio may be determined using the following formula: In the above formula, high_light_w may represent the target highlight ratio, and valid_blk_num may represent the total number of first image blocks, that is, the total number of valid image blocks in the valid field of view.

[0087] Step S16: Determine the target overexposure weight based on the target highlight ratio. For example, the larger the target highlight ratio is, the larger the target overexposure weight is. For example, a mapping relationship (also called a function relationship, a curve relationship, etc.) can be pre-configured, and the target overexposure weight is obtained by querying the mapping relationship based on the target highlight ratio.

[0088] In a possible implementation, multiple threshold intervals may be preconfigured, and different threshold intervals correspond to different mapping relationships, and the mapping relationship represents the relationship between the highlight ratio and the overexposure weight. Based on this, after obtaining the target highlight ratio, the target overexposure weight corresponding to the target highlight ratio may be obtained by querying the mapping relationship corresponding to the threshold interval to which the target highlight ratio belongs. For example, taking the configuration of three threshold intervals as an example, the three threshold intervals may be [0, size1), [size1, size2) and [size2, 1], respectively, where size1 and size2 represent configured values, size1 is greater than 0 and less than size2, and size2 is less than 1. Of course, the number of threshold intervals may be more or less.

[0089] On this basis, if the target highlight ratio is within the threshold interval [0, size1), the following formula can be used to determine the target overexposure weight: Alternatively, if the target highlight ratio is within the threshold interval [size1, size2), the following formula can be used to determine the target overexposure weight: Or, if the target highlight ratio is within the threshold interval [size2,1], the following formula is used to determine the target overexposure weight:

[0090] In the above formula, high_light_blk_w represents the target overexposure weight, high_light_w represents the target highlight ratio, and w1, w2, and w3 represent configured values, which can be configured based on experience.

[0091] To sum up, the target overexposure weight high_light_blk_w is proportional to the target highlight ratio high_light_w in a segmented form, that is, the more overexposed areas there are, the greater the target overexposure weight.

[0092] Step S17, determining a first average brightness based on the block average brightness of non-highlight blocks in the first image block, the block average brightness of highlight blocks in the first image block, the target overexposure weight, and the total number of first image blocks.

[0093] In a possible implementation, the following formula (1) may be used to determine the first average brightness. Of course, the following formula (1) is only an example, and the method for determining the first average brightness is not limited.

[0094]

[0095] In formula (1), y_ave may represent the first average brightness, valid_blk_num may represent the total number of first image blocks, high_light_blk_w may represent the target overexposure weight, K1 may represent the sum of the block average brightness of all non-highlight blocks, and K2 may represent the sum of the block average brightness of all highlight blocks. Referring to step S15, for each first image block, it can be known whether the first image block belongs to a highlight block or a non-highlight block. Based on this, the sum K1 of the block average brightness of all non-highlight blocks, i.e., the total brightness of non-highlight blocks, can be counted, and the sum K2 of the block average brightness of all highlight blocks, i.e., the total brightness of highlight blocks, can be counted.

[0096] As shown in formula (1), high_light_blk_w is the weight of the sum of the block average brightness of all highlight blocks. Therefore, the target overexposure weight can represent the weight corresponding to the block average brightness of the highlight block.

[0097] For example, in endoscope application scenarios, bright reflective plaques or spots are prone to appear. These plaques or spots may be reflections of medical devices, or they may be reflections of water droplets or bleeding spots. If these bright areas are referred to when calculating the first average brightness, the final exposure image may be darker. Therefore, it is necessary to first suppress the brightness of these highlight areas (that is, suppress them through high_light_blk_w×K2), that is, overexposure suppression, and obtain the first average brightness after overexposure suppression, so as to ensure that the brightness of the final exposure image is normal. At this point, the first average brightness of the first current frame image is obtained. Similarly, based on the second current frame image, the second average brightness of the second current frame image can also be obtained, see steps S11-S17.

[0098] Step 403, image brightness estimation. In the process of image brightness estimation, the first average brightness can be corrected based on the brightness information of the historical frame image to obtain the brightness estimation value of the first current frame image. The purpose of the brightness estimation value is to make the image brightness reach the target brightness as quickly and stably as possible, which is a correction to the first average brightness. For example, if the first current frame image is a first white light image, then the historical frame image can be a historical frame white light image, and the next frame image can be a next frame white light image. Alternatively, if the first current frame image is a first fluorescent image, then the historical frame image can be a historical frame fluorescent image, and the next frame image can be a next frame fluorescent image.

[0099] For example, after obtaining the first average brightness of the first current frame image, the first average brightness may not be accurate. In order to improve the accuracy of subsequent processing and reduce the delay of subsequent processing, the first average brightness can also be corrected based on the brightness information of the historical frame image to obtain a corrected brightness estimate.

[0100] Exemplarily, the brightness information of the historical frame image may include the average brightness of the historical frame image and the brightness estimation value of the historical frame image. For example, when the first current frame image is used as the historical frame image, the first average brightness of the first current frame image is the average brightness of the historical frame image, and the brightness estimation value of the first current frame image is the brightness estimation value of the historical frame image. For example, the image brightness estimation process may include:

[0101] Step S21, determining a brightness prediction value of the first current frame image based on the average brightness of the historical frame images, where the brightness prediction value is the average brightness of the first current frame image predicted using the average brightness of the historical frame images.

[0102] Exemplarily, the brightness variation law of multiple frames of images can be expressed by y=kt, where t represents time, the image sequence number can be used as time, k represents the slope, and y represents the average brightness (i.e., brightness prediction value). The brightness variation law can be obtained by using the average brightness of two historical frame images, that is, the value of the slope k can be obtained.

[0103] For example, the second historical frame image before the first current frame image is called historical frame image p1, and the first historical frame image before the first current frame image is called historical frame image p2. The time t of historical frame image p1 is 1 (indicating that the image sequence number is 1), and the average brightness y of historical frame image p1 is y1. The time t of historical frame image p2 is 2, and the average brightness y of historical frame image p2 is y2. Based on this, y1=k*1, y2=k*2, that is, k=y2-y1, and k can be the brightness change law of multiple frame images. The time t of the first current frame image is 3, so the average brightness y3 (that is, k*3) of the first current frame image can be obtained, and the average brightness y3 of the first current frame image is the brightness prediction value of the first current frame image (recorded as y_pre).

[0104] Step S22: Perform a weighted operation on the first average brightness of the first current frame image and the brightness prediction value of the first current frame image to obtain an average brightness correction value of the first current frame image. For example, the average brightness correction value can be obtained by using the following formula: y_ave_1 = a 1 *y_ave+a 2 *y_pre.a 1 and a 2 Represents the weighting coefficient, which can be configured based on experience, such as a 1 +a 2 =1, for this 1 and a 2 There is no restriction on the value of y_ave. y_ave_1 represents the first average brightness, y_pre represents the brightness prediction value, and y_ave_2 represents the average brightness correction value.

[0105] Step S23: Perform a weighted operation on the brightness estimation value of the historical frame image and the average brightness correction value of the first current frame image to obtain the brightness estimation value of the first current frame image. For example, the brightness estimation value can be obtained by using the following formula: y_est = b 1 *y_ave_1+b 2 *y_est_last.b 1 and b 2 Represents the weighting coefficient, which can be configured based on experience, such as b 1 +b 2 =1, for this b 1 and b 2 There is no restriction on the value of y_est_last represents the brightness estimation value of the historical frame image, and y_est represents the brightness estimation value of the first current frame image.

[0106] For example, in order to prevent flickering, jittering and the like in the final exposed image, after obtaining the average brightness correction value y_ave_1, the average brightness correction value y_ave_1 can also be subjected to time domain smoothing processing, that is, the average brightness correction value y_ave_1 is subjected to time domain smoothing processing using the brightness estimation value y_est_last of the historical frame image, referring to the time domain smoothing processing of the above formula, to obtain the brightness estimation value y_est of the first current frame image.

[0107] At this point, step 403 is completed, and the brightness estimation value y_est of the first current frame image is obtained.

[0108] Step 404: Determine a first difference between the target brightness and the brightness estimation value. For example, the target brightness is subtracted from the brightness estimation value to obtain the first difference, that is, the target brightness minus the brightness estimation value.

[0109] Exemplarily, the target brightness may be a brightness value preconfigured based on experience, or may be a default brightness value. The target brightness is the brightness that the user wants the image to achieve, and there is no restriction on the target brightness.

[0110] If the absolute value of the first difference is less than a preset threshold value (which can be configured based on experience and is a value close to 0, such as 0.5, 0.8, etc.), it means that the control target has been achieved, that is, the exposure parameters of the first current frame image have met expectations, and the exposure parameters of the next frame image may not be changed. Therefore, the process can be terminated without executing subsequent steps, that is, the exposure parameters of the next frame image are not adjusted. If the absolute value of the first difference is not less than the preset threshold value, it means that the control target has not been achieved, that is, the exposure parameters of the first current frame image do not meet expectations, and the exposure parameters of the next frame image need to be changed. Therefore, subsequent steps can be executed.

[0111] Step 405, calculation of expected brightness change multiple. In the process of calculating the expected brightness change multiple, the target change multiple of the first current frame image can be determined based on the first difference and the second difference (i.e., the difference between the target brightness and the brightness estimation value of the historical frame image), and the target change multiple represents the expected change ratio when changing from the first average brightness (i.e., the first average brightness of the first current frame image) to the target brightness.

[0112] Exemplarily, based on a first difference between the target brightness and the brightness estimation value of the first current frame image, and a second difference between the target brightness and the brightness estimation value of the historical frame image, segmented PID (proportional integral differential) control can be performed to obtain the expected change multiple of the brightness, which is recorded as the target change multiple.

[0113] For example, the calculation process of the expected brightness change multiple may include the following steps:

[0114] Step S31: Determine the brightness control amount corresponding to the brightness estimated value of the first current frame image based on the first difference, the second difference, the configured proportional adjustment coefficient, and the configured differential adjustment coefficient.

[0115] Exemplarily, piecewise PID control can be performed on two cases, such as the image changing from bright to dark and the image changing from dark to bright, that is, different PID control methods are used in different cases, so as to obtain a more accurate brightness control amount.

[0116] For example, if y_ref ≥ y_est, where y_ref represents the target brightness and y_est represents the brightness estimated value of the first current frame image, the brightness control amount can be determined using the following formula: pid_out = kp 1 *y_err + kd 1 *(y_err - y_last_err). For example, y_err represents the first difference, y_last_err represents the second difference, kp 1 represents the configured proportional adjustment coefficient, which can be configured according to experience and there is no limit to this proportional adjustment coefficient, kd 1 represents the configured differential adjustment coefficient, which can be configured according to experience and there is no limit to this differential adjustment coefficient, and pid_out represents the brightness control amount corresponding to the brightness estimated value.

[0117] If y_ref < y_est, the brightness control amount can be determined using the following formula: pid_out = kp 2 *y_err + kd 2 *(y_err - y_last_err). For example, kp 2 can represent the configured proportional adjustment coefficient, which can be configured according to experience and there is no limit to this proportional adjustment coefficient, kd 2 can represent the configured differential adjustment coefficient, which can be configured according to experience and there is no limit to this differential adjustment coefficient.

[0118] Step S32: Determine the initial change multiple of the first current frame image based on the brightness control amount and the brightness estimated value of the first current frame image, and the initial change multiple can be a multiple value between 0 and 2.

[0119] For example, the initial change multiple can be determined using the following formula: pid out represents the brightness control amount, which can be a positive value or a negative value, y est represents the brightness estimated value of the first current frame image. If is less than -1, it is truncated so that equals -1. If is greater than 1, it is truncated so that is equal to 1, so is a value between -1 and 1, i.e. gain is a value in [0,2], ae gain Indicates the initial change factor.

[0120] Step S33: Perform a weighted operation on the initial change multiple of the first current frame image and the target change multiple of the historical frame image to obtain the target change multiple of the first current frame image. For example, the target change multiple of the first current frame image can be obtained by the following formula: ae_gain_1 = c 1 *ae_gain+c 2 *ae_gain_last. c 1 and c 2 Represents the weighting coefficient, which can be configured based on experience, such as c 1 +c 2 =1, for which c 1 and c 2 There is no restriction on the value of ae_gain represents the initial change multiple of the first current frame image, ae_gain_last represents the target change multiple of the historical frame image, and ae_gain_1 represents the target change multiple of the first current frame image.

[0121] For example, in order to prevent flickering, jittering and the like in the final exposure image, after obtaining the initial change multiple ae_gain, the initial change multiple ae_gain can also be subjected to time domain smoothing processing, that is, the target change multiple ae_gain_last of the historical frame image is used to perform time domain smoothing processing on the initial change multiple ae_gain, referring to the time domain smoothing processing of the above formula to obtain the target change multiple of the first current frame image.

[0122] At this point, the calculation process of the expected brightness change multiple is completed, and the target change multiple is obtained.

[0123] Step 406, exposure parameter calculation. In the exposure parameter calculation process, the first exposure parameter of the next frame image can be determined based on the target change multiple of the first current frame image and the exposure parameter of the first current frame image (i.e., the current exposure parameter). The first exposure parameter may include, but is not limited to, at least one of: light source intensity, sensor gain, and first algorithm digital gain. For example, if the first current frame image is a first white light image, the first exposure parameter is the exposure parameter of the next white light image, that is, the first exposure parameter is used to capture the next white light image. If the first current frame image is a first fluorescent image, the first exposure parameter is the exposure parameter of the next fluorescent image, that is, the first exposure parameter is used to capture the next fluorescent image.

[0124] Exemplarily, the target change multiple can be a multiple value between 0 and 2. If the target change multiple is greater than a preset change multiple threshold (such as 1), the light source intensity in the exposure parameters of the first current frame image is increased (such as the light source intensity multiplied by the target change multiple) to obtain the light source intensity in the first exposure parameter (i.e., the value multiplied by the target change multiple). And / or, the sensor gain in the exposure parameters of the first current frame image is increased to obtain the sensor gain in the first exposure parameter (such as the current sensor gain multiplied by the target change multiple to obtain the changed sensor gain, and the changed sensor gain is used as the first exposure parameter). And / or, the first algorithm digital gain in the exposure parameters of the first current frame image is increased to obtain the first algorithm digital gain in the first exposure parameter (such as the current first algorithm digital gain multiplied by the target change multiple to obtain the changed first algorithm digital gain, and the changed first algorithm digital gain is used as the first exposure parameter).

[0125] Exemplarily, if the target change multiple is less than a preset change multiple threshold (such as 1), the light source intensity in the exposure parameters of the first current frame image is reduced to obtain the light source intensity in the first exposure parameters (such as the current light source intensity multiplied by the target change multiple). And / or, the sensor gain in the exposure parameters of the first current frame image is reduced to obtain the sensor gain in the first exposure parameters. And / or, the first algorithm digital gain in the exposure parameters of the first current frame image is reduced to obtain the first algorithm digital gain in the first exposure parameters.

[0126] In a possible implementation, a method of automatically adjusting parameters according to ambient brightness to obtain an image with appropriate brightness is called automatic exposure. Exposure parameters involved in the automatic exposure include shutter, gain, and aperture.

[0127] For the shutter (shutter time), the exposure time refers to the time interval from the opening to the closing of the shutter. When the exposure time increases, the image brightness increases, and when the exposure time is short, the image brightness also decreases. The shutter speed can be 8 seconds, 4 seconds, 2 seconds, 1 second, 1 / 2 second, 1 / 4 second, 1 / 8 second, 1 / 15 second, 1 / 30 second, 1 / 60 second, 1 / 125 second, 1 / 250 second, 1 / 500 second, 1 / 1000 second, 1 / 2000 second, 1 / 4000 second, etc. Under the stroboscopic light source, in order to prevent image flickering and image light and dark stratification, the shutter time of the first sensor and the second sensor is fixed according to the blanking and clock cycle of the sensor, and there is no restriction on the method of determining the shutter time. For the aperture, considering that the endoscope is an electronic scope, the aperture size has been determined in the design stage of the endoscope and is not arbitrarily configured in the later stage. Therefore, the aperture does not need to be adjusted.

[0128] With respect to gain, gain can be divided into sensor gain and algorithm digital gain. The first sensor and the second sensor can use the same sensor gain. When obtaining the first current frame image, ISP can also be performed on the first current frame image. ISP can include but is not limited to: bad pixel correction, black level correction, 2D noise reduction, de-mosaicing, color correction, sharpening, tone mapping and other algorithms. When performing ISP on the first current frame image, it is necessary to use algorithm digital gain to perform ISP on the first current frame image, and the algorithm digital gain is referred to as the first algorithm digital gain. When obtaining the second current frame image, it is also possible to use algorithm digital gain to perform ISP on the second current frame image, and the algorithm digital gain is referred to as the second algorithm digital gain.

[0129] Regarding sensor gain, the first sensor (second sensor) has sensitivity, which is the gain applied by the system to the electrical signal. The analog electrical signal converted by the first sensor will pass through the automatic gain control (AGC) module before being converted into a digital signal. This module amplifies the input signal so that the output signal intensity meets the final requirements for image brightness. Adjusting the sensor gain is particularly effective when the incident light capacity is small and the aperture and exposure time cannot meet the exposure standard requirements. Regarding the algorithm digital gain, when performing ISP on the first current frame image (second current frame image), the algorithm digital gain can improve the quality of the first current frame image and achieve better image effects.

[0130] Regarding the intensity of the light source, if the white fluorescent stroboscopic mode does not require a linked light source, overexposure is likely to occur, which greatly affects the doctor's visual perception and may endanger the life safety of the target object to a certain extent. Therefore, an exposure method using a linked light source is adopted, that is, it is necessary to control the light source intensity of the light source and use the light source intensity as an exposure parameter to ensure the practicality and stability of the exposure function, avoid overexposure, and improve the user experience of medical staff.

[0131] In summary, the first exposure parameter may include light source intensity, sensor gain and first algorithm digital gain. The first algorithm digital gain may also be referred to as ISP gain, which is used to perform ISP on the first current frame image.

[0132] Considering that the imaging noise should be as small as possible and the stroboscopic light source brightness itself is relatively small, therefore, if the image brightness needs to be enhanced (i.e. the target change multiple is greater than the preset change multiple threshold), the light source intensity is increased first, then the sensor gain is increased, and then the first algorithm digital gain is increased. For example, first determine whether the current light source intensity reaches the maximum light source intensity. If the maximum light source intensity is not reached, increase the light source intensity (the light source intensity after the increase cannot exceed the maximum light source intensity), and use the increased light source intensity as the first exposure parameter. In addition, use the current sensor gain and the current first algorithm digital gain as the first exposure parameter.

[0133] If the maximum light source intensity is reached, it is determined whether the current sensor gain reaches the maximum sensor gain. If the maximum sensor gain is not reached, the sensor gain is increased (the increased sensor gain cannot exceed the maximum sensor gain), and the increased sensor gain is used as the first exposure parameter. In addition, the current light source intensity (i.e., the maximum light source intensity) and the current first algorithm digital gain are used as the first exposure parameter.

[0134] If the maximum sensor gain is reached, it is determined whether the current first algorithm digital gain reaches the maximum first algorithm digital gain. If not, the first algorithm digital gain is increased, and the increased first algorithm digital gain is used as the first exposure parameter. The current light source intensity and the current sensor gain are used as the first exposure parameter.

[0135] Exemplarily, if the image brightness needs to be reduced (i.e., the target change multiple is less than the preset change multiple threshold), the first algorithm digital gain is first reduced, then the sensor gain is reduced, and then the light source intensity is reduced. For example, first determine whether the current first algorithm digital gain reaches the minimum first algorithm digital gain, if not, reduce the first algorithm digital gain, if yes, determine whether the current sensor gain reaches the minimum sensor gain, if not, reduce the sensor gain, if yes, determine whether the current light source intensity reaches the minimum light source intensity, if not, reduce the light source intensity, this process is similar to the process of enhancing the image brightness, and will not be repeated.

[0136] Exemplarily, after obtaining the sensor gain, the sensor gain can be input to the first sensor and the second sensor, and the first sensor captures the next frame of image based on the sensor gain, and the second sensor captures the next frame of image based on the sensor gain. There is no restriction on this image acquisition process.

[0137] Exemplarily, after obtaining the first algorithm digital gain, the first algorithm digital gain can be input to the ISP platform (i.e., the entity used to perform ISP processing on the image), and the ISP platform performs ISP processing on the next frame of the image based on the first algorithm digital gain, and there is no restriction on this ISP processing process.

[0138] Exemplarily, after the light source intensity is obtained, the light source intensity may be input to the light source, and the light source emits light of appropriate intensity based on the light source intensity, that is, emits light of appropriate intensity for the next frame of image.

[0139] When the light source intensity is input to the light source, the light source intensity can also be encoded to obtain a light source intensity encoding value, and the light source intensity encoding value is input to the light source, and the light source can decode the light source intensity encoding value accordingly to obtain the light source intensity. For example, the light source intensity may be a decimal, such as 1.5, etc. Encoding the light source intensity means converting the light source intensity into an integer, such as multiplying the light source intensity by 10 to obtain the light source intensity encoding value. Correspondingly, decoding the light source intensity encoding value means dividing the light source intensity encoding value by 10.

[0140] When the light source emits light of appropriate intensity based on the light source intensity, the mapping relationship (such as the mapping relationship between the light source intensity and the light source control parameters) can be queried through the light source intensity to obtain the light source control parameters corresponding to the light source intensity, and then the light source emits light of appropriate intensity based on the light source control parameters.

[0141] The mapping relationship between the light source intensity (luminous flux) and the light source control parameter can be calibrated in advance. For example, when the light source emits light based on the light source control parameter d1, if the light source intensity is the light source intensity e1, then the mapping relationship between the light source intensity e1 and the light source control parameter d1 is recorded, and so on.

[0142] After the mapping relationship is obtained, the mapping relationship may be integrated into the light source, so that the mapping relationship may be queried through the intensity of the light source to obtain the light source control parameter corresponding to the intensity of the light source.

[0143] Regarding the light source control parameter, the light source control parameter may be current, the light source control parameter may be voltage, or other parameters, which are not limited.

[0144] Step 407, brightness ratio calculation. For example, since the first sensor and the second sensor are exposed separately, that is, the first current frame image is obtained when the first sensor is exposed, and the second current frame image is obtained when the second sensor is exposed, in order to prevent the exposure process and exposure result from being inconsistent and to prevent the three-dimensional effect from being affected, in the brightness ratio calculation process, the second exposure parameter of the next frame image can be determined based on the first average brightness of the first current frame image, the second average brightness of the second current frame image and the first algorithm digital gain, and the second exposure parameter can include but is not limited to the second algorithm digital gain, which is the algorithm digital gain used to perform ISP processing on the second current frame image.

[0145] Exemplarily, after obtaining the second algorithm digital gain, the second algorithm digital gain may be input to the ISP platform, and the ISP platform performs ISP processing on the next frame of image based on the second algorithm digital gain.

[0146] Exemplarily, for the brightness ratio calculation process, the brightness ratio calculation process may include:

[0147] Step S41: Determine the quotient between the second average brightness and the first average brightness. For example, the quotient can be y_stat_right represents the second average brightness, and y_stat_left represents the first average brightness.

[0148] Step S42: If the quotient is less than a preset ratio threshold (y_stat_thr), the digital gain of the second algorithm is determined based on the product of the digital gain of the first algorithm and the quotient. For example, the digital gain of the second algorithm can be determined using the following formula: For example, y_dgain_right represents the digital gain of the second algorithm, and y_dgain_left represents the digital gain of the first algorithm.

[0149] Step S43: If the quotient is not less than the preset ratio threshold (y_stat_thr), the first algorithm digital gain is determined as the second algorithm digital gain, that is, y_dgain_right=y_dgain_left.

[0150] At this point, the image processing method in the white fluorescent stroboscopic mode is completed.

[0151] Case 2: Image processing method in white light mode.

[0152] See also Figure 6 FIG. 1 is a flow chart of an image processing method in a white light mode. The method can be applied to a camera system host (such as an image processing unit). The process may include:

[0153] Step 601: Acquire a first current frame image and a second current frame image.

[0154] For example, in the white light mode, the light source can continuously output white light, the image sensor can sense the ambient light, obtain a white light image, and send the white light image to the camera system host. For example, when the light source outputs white light, the third white light image can be collected by the first sensor (to distinguish it from the first white light image), and the fourth white light image can be collected by the second sensor (to distinguish it from the second white light image).

[0155] Based on this, when the camera system host obtains the first current frame image and the second current frame image, the first current frame image may be the third white light image, and the second current frame image may be the fourth white light image.

[0156] Step 602, image brightness statistics. In the image brightness statistics process, a third average brightness of the third white light image can be determined, and the third average brightness can be determined based on the brightness value of each pixel of the third white light image. In addition, a fourth average brightness of the fourth white light image can be determined, and the fourth average brightness can be determined based on the brightness value of each pixel of the fourth white light image. For example, regarding the determination method of the third average brightness and the fourth average brightness, please refer to step 402, which will not be repeated here.

[0157] Step 603, image brightness estimation. In the process of image brightness estimation, the third average brightness can be corrected based on the brightness information of the historical frame white light image to obtain a brightness estimation value of the third white light image. The purpose of the brightness estimation value is to make the image brightness reach the target brightness as quickly and stably as possible, which is a correction to the third average brightness. For example, the method for determining the brightness estimation value of the third white light image can be referred to step 403, which will not be repeated here.

[0158] Step 604: Determine a first difference between the target brightness and the brightness estimation value. For example, the target brightness is subtracted from the brightness estimation value to obtain the first difference, that is, the target brightness minus the brightness estimation value.

[0159] Step 605, calculation of the expected brightness change multiple. In the process of calculating the expected brightness change multiple, the target change multiple of the third white light image can be determined based on the first difference and the second difference (i.e., the difference between the target brightness and the brightness estimation value of the historical frame white light image), and the target change multiple represents the expected change ratio when the third average brightness changes to the target brightness. For example, the method for determining the target change multiple of the third white light image can be referred to step 405, which will not be repeated here.

[0160] Step 606, exposure parameter calculation. In the exposure parameter calculation process, the third exposure parameter of the next frame of white light image can be determined based on the target change multiple of the third white light image and the exposure parameter of the third white light image (current exposure parameter) (i.e., the next frame of white light image is captured based on the third exposure parameter). The third exposure parameter may include but is not limited to: at least one of shutter time, sensor gain, and third algorithm digital gain. Obviously, compared with the first exposure parameter, the third exposure parameter has an additional shutter time, and the third exposure parameter does not have light source intensity, that is, the exposure process in the white light mode does not require linkage of the light source. For example, regarding the method of determining the third exposure parameter, please refer to step 406, except that the light source intensity is replaced by shutter time. The other processes are similar to the determination of the first exposure parameter, and will not be repeated here.

[0161] Step 607, brightness ratio calculation. In the brightness ratio calculation process, the fourth exposure parameter of the next frame of white light image can be determined based on the third average brightness of the third white light image, the fourth average brightness of the fourth white light image, and the third algorithm digital gain, and the fourth exposure parameter can include but is not limited to the fourth algorithm digital gain. The implementation method of step 607 can refer to step 407, which will not be repeated here.

[0162] At this point, the image processing method in the white light mode is completed.

[0163] Case 3: Image processing method during mode switching.

[0164] When switching between the two exposure modes, abnormal phenomena are prone to occur. For example, due to the long transmission link of the light source signal, when the exposure mode is switched, there is an indefinite delay in the light source mode switching, and then a short overexposure problem occurs when the white light mode is switched to the white fluorescent stroboscopic mode, and a short purple screen phenomenon occurs when the white fluorescent stroboscopic mode is switched to the white light mode. In response to the above findings, this embodiment adopts the following processing:

[0165] If the endoscope is switched from white light mode to white fluorescent stroboscopic mode:

[0166] After receiving the instruction to switch from the white light mode to the white fluorescent stroboscopic mode, the mode switching process is triggered. In the mode switching process, before the first current frame image and the second current frame image are acquired for the first time, the first exposure parameter is set to the first fixed exposure parameter, and the second exposure parameter is set to the second fixed exposure parameter. In this case, the first current frame image and the second current frame image can be acquired.

[0167] Each time the first current frame image and the second current frame image are acquired, case 1 can be executed to obtain the first exposure parameter and the second exposure parameter. It should be noted that after obtaining the first exposure parameter and the second exposure parameter, the first exposure parameter and the second exposure parameter are not directly output, but the first exposure parameter and the second exposure parameter need to be updated. For example, the update can be performed in the following manner:

[0168] When the first current frame image and the second current frame image are acquired for the Mth time, the correction coefficient corresponding to M can be determined, wherein the larger M is, the larger the correction coefficient is. Based on this, after obtaining the first exposure parameter and the second exposure parameter, the updated first exposure parameter can be determined based on the first exposure parameter and the correction coefficient, such as the product value of the two as the updated first exposure parameter. The updated second exposure parameter can be determined based on the second exposure parameter and the correction coefficient, such as the product value of the two as the updated second exposure parameter. On this basis, the updated first exposure parameter and the updated second exposure parameter are output.

[0169] Regarding the correction coefficient corresponding to M, the total number of frames A of the switching process can be obtained in advance. The total number of frames A represents the number of frames where overexposure occurs according to experimental statistics during the process of switching from the white light mode to the white fluorescent stroboscopic mode (a small positive tolerance can be added to this number of frames), that is, there are a total of A frames of images with overexposure.

[0170] Then, the ratio of M to A can be used as a correction coefficient. For example, if A is 10, when the first current frame image and the second current frame image are acquired for the first time, the correction coefficient is 0.1; when the first current frame image and the second current frame image are acquired for the second time, the correction coefficient is 0.2, and so on.

[0171] Obviously, when the first current frame image and the second current frame image are acquired for the tenth time, the correction coefficient is 1. At this point, the transition in the mode switching process is completed, and the mode switching process is completed. Case 1 can be adopted.

[0172] To sum up, during the mode switching process, the first exposure parameter is first set to the first fixed exposure parameter, and the second exposure parameter is set to the second fixed exposure parameter to make the image darker. Then, as the number of frames increases, the correction coefficients of the first exposure parameter and the second exposure parameter are linearly increased until the correction coefficient is 1, and the first exposure parameter and the second exposure parameter are restored to normal.

[0173] If the endoscope is switched from white fluorescent stroboscopic mode to white light mode:

[0174] After receiving an instruction to switch from the white fluorescent stroboscopic mode to the white light mode, the mode switching process is triggered. During the mode switching process, before acquiring the third white light image and the fourth white light image for the first time, the third exposure parameter may be set to the third fixed exposure parameter, and the fourth exposure parameter may be set to the fourth fixed exposure parameter. In this case, the third white light image and the fourth white light image may be acquired.

[0175] Each time the third white light image and the fourth white light image are acquired, the case 2 can be executed to obtain the third exposure parameter and the fourth exposure parameter. It should be noted that after obtaining the third exposure parameter and the fourth exposure parameter, the third exposure parameter and the fourth exposure parameter are not directly output, but the third exposure parameter and the fourth exposure parameter need to be updated. For example, the update can be performed in the following manner:

[0176] When the third white light image and the fourth white light image are acquired for the Nth time, the correction coefficient corresponding to N is determined, and N can be a positive integer; wherein the larger N is, the larger the correction coefficient is. Based on this, after obtaining the third exposure parameter and the fourth exposure parameter, the updated third exposure parameter is determined based on the third exposure parameter and the correction coefficient, such as the product value of the two is used as the updated third exposure parameter. The updated fourth exposure parameter is determined based on the fourth exposure parameter and the correction coefficient, such as the product value of the two is used as the updated fourth exposure parameter. On this basis, the updated third exposure parameter and the updated fourth exposure parameter are output.

[0177] Regarding the correction coefficient corresponding to N, the total number of frames B of the switching process can be obtained in advance. The total number of frames B represents the number of frames where the purple screen phenomenon occurs according to experimental statistics during the process of switching from the white fluorescent stroboscopic mode to the white light mode (a small positive tolerance can be added to this number of frames). That is, assuming that B is 5, then, during the process of switching from the white fluorescent stroboscopic mode to the white light mode, starting from the first frame image at the beginning of the mode switching, there are a total of 5 frames of images with the purple screen phenomenon, that is, there are a total of 5 frames of images with abnormalities.

[0178] Then, the ratio of N to B is used as a correction coefficient. Obviously, when the third white light image and the fourth white light image are acquired for the Bth time, the correction coefficient is 1, and the mode switching process is completed. Case 2 can be adopted.

[0179] It can be seen from the above technical solutions that in the embodiment of the present application, only two image sensors need to be deployed inside the endoscope at the same time, saving the internal space of the endoscope. The light source alternately outputs white light and fluorescence, so that the endoscope supports three-dimensional white light images and three-dimensional fluorescence images almost simultaneously. The exposure parameters (such as light source intensity) of the next frame image (white light image or fluorescence image) can be controlled, thereby linking the light source of the endoscope, that is, controlling the light source intensity of the light source, ensuring the practicality and stability of the exposure function, and not prone to overexposure, thereby improving the user experience of medical staff. For the white fluorescence stroboscopic mode, combined with the characteristics of medical images and three-dimensional images, the image brightness estimation, brightness expected change multiple calculation and brightness ratio calculation processes are designed in a targeted manner, and an automatic exposure method of the linkage light source suitable for medical scenes is given, which ensures the practicality and stability of the exposure function of the 3D fluorescence device. For the white light mode, combined with the characteristics of medical images and three-dimensional images, the image brightness estimation, brightness expected change multiple calculation and brightness ratio calculation are designed in a targeted manner, and an automatic exposure method suitable for medical scenes is given, which ensures the practicality and stability of the exposure function of the 3D fluorescence device. For the mode switching process, for the endoscope system with multi-mode exposure, the exposure strategy is optimized to target the abnormal phenomena caused by mode switching and reduce the frequency and duration of the abnormal phenomena.

[0180] Based on the same application concept as the above method, an image processing device for a 3D endoscope is proposed in an embodiment of the present application. If the 3D endoscope adopts a white fluorescence stroboscopic mode, white light and fluorescence are alternately output by a light source; when the light source outputs white light, a first white light image is collected by a first sensor, and a second white light image is collected by a second sensor; when the light source outputs fluorescence, a first fluorescence image is collected by the first sensor, and a second fluorescence image is collected by the second sensor, see Figure 7 FIG. 1 is a schematic diagram of the structure of the device, wherein the device comprises:

[0181] An acquisition module 71 is used to acquire a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image;

[0182] A determination module 72 is configured to determine a first average brightness of a first current frame image and a second average brightness of a second current frame image, wherein the first average brightness is determined based on the brightness value of each pixel of the first current frame image; and determine a target change multiple based on the first average brightness and a configured target brightness, wherein the target change multiple represents an expected change ratio when the first average brightness changes to the target brightness;

[0183] A control module 73 is configured to determine a first exposure parameter based on the target change multiple and an exposure parameter of a first current frame image, wherein the first exposure parameter includes at least one of light source intensity, sensor gain, and a first algorithm digital gain; and determine a second exposure parameter based on the first average brightness, the second average brightness, and the first algorithm digital gain, wherein the second exposure parameter includes a second algorithm digital gain;

[0184] Among them, if the first current frame image is a first white light image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame white light image; if the first current frame image is a first fluorescent image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame fluorescent image.

[0185] Exemplarily, when determining the first average brightness of the first current frame image, the determination module 72 is specifically used to: divide the first current frame image into multiple image blocks, and for each image block, determine the block average brightness of the image block based on the brightness value of each pixel point of the image block; obtain a first image block located in an effective field of view area and a second image block not located in the effective field of view area, wherein the effective field of view area is an effective illumination area of ​​the light source; determine the first average brightness based on the block average brightness of non-highlight blocks in the first image block, the block average brightness of highlighted blocks in the first image block, the acquired target overexposure weight and the total number of first image blocks; the target overexposure weight represents the weight corresponding to the block average brightness of the highlighted blocks.

[0186] Exemplarily, the determination module 72 is further used to determine a brightness threshold based on the ratio of the central area brightness of the first current frame image and the configured brightness threshold, wherein the central area brightness is the average brightness value of each image block in the central area; determine a first boundary and a second boundary of the first dimension, and determine a third boundary and a fourth boundary of the second dimension based on the first boundary, the second boundary, and the width and height of the first current frame image; wherein, if the first dimension is a column dimension and the second dimension is a row dimension, the first boundary is the first column that meets the preset condition from the left, and the second boundary is the first column that meets the preset condition from the right; if the first dimension is a row dimension and the second dimension is a column dimension, the first boundary is the first row that meets the preset condition from the top, and the second boundary is the first row that meets the preset condition from the bottom; a column that meets the preset condition means that the average value of the block average brightness of all image blocks in the column is not less than the brightness threshold; a row that meets the preset condition means that the average value of the block average brightness of all image blocks in the row is not less than the brightness threshold; wherein, if the image block is located in the area formed by the first boundary, the second boundary, the third boundary and the fourth boundary, the image block is the first image block; otherwise, the image block is the second image block.

[0187] Exemplarily, the determination module 72 is also used to determine the target overexposure weight; when the determination module 72 determines the target overexposure weight, it is specifically used to: determine the target highlight ratio based on the total number of highlight blocks and the total number of first image blocks; wherein, for each first image block, if the block average brightness of the first image block is not less than the highlight block brightness threshold, then the first image block belongs to the highlight block, if the block average brightness of the first image block is less than the highlight block brightness threshold, then the first image block does not belong to the highlight block; determine the target overexposure weight based on the target highlight ratio; wherein, the larger the target highlight ratio, the larger the target overexposure weight; wherein, multiple threshold intervals are pre-configured, different threshold intervals correspond to different mapping relationships, and the mapping relationship represents the relationship between the highlight ratio and the overexposure weight; the target overexposure weight is obtained by querying the mapping relationship corresponding to the threshold interval to which the target highlight ratio belongs.

[0188] Exemplarily, when the determination module 72 determines the target change multiple based on the first average brightness and the configured target brightness, it is specifically used to: correct the first average brightness based on the brightness information of the historical frame image to obtain a brightness estimation value of the first current frame image; determine a first difference between the target brightness and the brightness estimation value; determine the target change multiple of the first current frame image based on the first difference and the second difference; wherein the second difference is the difference between the target brightness and the brightness estimation value of the historical frame image; if the first current frame image is a first white light image, the historical frame image is a historical frame white light image, and if the first current frame image is a first fluorescent image, the historical frame image is a historical frame fluorescent image.

[0189] Exemplarily, the brightness information of the historical frame image includes the average brightness and the brightness estimation value of the historical frame image. The determination module 72 corrects the first average brightness based on the brightness information of the historical frame image to obtain the brightness estimation value of the first current frame image, and is specifically used for: determining the brightness prediction value of the first current frame image based on the average brightness of the historical frame image, performing a weighted operation on the first average brightness and the brightness prediction value to obtain an average brightness correction value; performing a weighted operation on the brightness estimation value of the historical frame image and the average brightness correction value to obtain the brightness estimation value of the first current frame image.

[0190] Exemplarily, when the determination module 72 determines the target change multiple of the first current frame image based on the first difference and the second difference, it is specifically used to: determine the brightness control amount corresponding to the brightness estimation value based on the first difference, the second difference, the configured proportional adjustment coefficient and the configured differential adjustment coefficient; determine the initial change multiple of the first current frame image based on the brightness control amount and the brightness estimation value; perform weighted operation on the initial change multiple of the first current frame image and the target change multiple of the historical frame image to obtain the target change multiple of the first current frame image.

[0191] Exemplarily, when determining the first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image, the control module 73 is specifically used to: if the target change multiple is greater than a preset change multiple threshold, increase the light source intensity in the exposure parameter of the first current frame image to obtain the light source intensity in the first exposure parameter; and / or, increase the sensor gain in the exposure parameter of the first current frame image to obtain the sensor gain in the first exposure parameter; and / or, increase the first algorithm digital gain in the exposure parameter of the first current frame image to obtain the first algorithm digital gain in the first exposure parameter; if the target change multiple is less than the preset change multiple threshold, reduce the light source intensity in the exposure parameter of the first current frame image to obtain the light source intensity in the first exposure parameter; and / or, reduce the sensor gain in the exposure parameter of the first current frame image to obtain the sensor gain in the first exposure parameter; and / or, reduce the first algorithm digital gain in the exposure parameter of the first current frame image to obtain the first algorithm digital gain in the first exposure parameter.

[0192] Exemplarily, when the control module 73 determines the second algorithm digital gain based on the first average brightness, the second average brightness and the first algorithm digital gain, it is specifically used to:

[0193] Determine a quotient between the second average brightness and the first average brightness; if the quotient is less than a preset ratio threshold, determine the second algorithm digital gain based on the product value between the first algorithm digital gain and the quotient; or, if the quotient is not less than a preset ratio threshold, determine the first algorithm digital gain as the second algorithm digital gain.

[0194] Exemplarily, if the 3D endoscope adopts the white light mode, white light is continuously output through the light source, a third white light image is collected through the first sensor, and a fourth white light image is collected through the second sensor; the determination module 72 is also used to determine the third average brightness of the third white light image and the fourth average brightness of the fourth white light image; determine the target change multiple based on the third average brightness and the target brightness; the control module 73 is also used to determine the third exposure parameter of the next frame of white light image based on the target change multiple and the exposure parameter of the third white light image, the third exposure parameter including at least one of the shutter time, sensor gain and the third algorithm digital gain; determine the fourth exposure parameter of the next frame of white light image based on the third average brightness, the fourth average brightness and the third algorithm digital gain, the fourth exposure parameter including the fourth algorithm digital gain.

[0195] Exemplarily, if the 3D endoscope switches from the white light mode to the white fluorescent stroboscopic mode, the determination module 72 is also used to, for the mode switching process, set the first exposure parameter to the first fixed exposure parameter and the second exposure parameter to the second fixed exposure parameter before acquiring the first current frame image and the second current frame image for the first time; when the first current frame image and the second current frame image are acquired for the Mth time, determine the correction coefficient corresponding to M; wherein, the larger M is, the larger the correction coefficient is, and M can be a positive integer; the control module 73 is also used to determine the updated first exposure parameter based on the first exposure parameter and the correction coefficient after determining the first exposure parameter and the second exposure parameter; and determine the updated second exposure parameter based on the second exposure parameter and the correction coefficient.

[0196] Exemplarily, if the 3D endoscope switches from the white fluorescence stroboscopic mode to the white light mode, the determination module 72 is also used to, for the mode switching process, set the third exposure parameter to the third fixed exposure parameter and the fourth exposure parameter to the fourth fixed exposure parameter before acquiring the third white light image and the fourth white light image for the first time; when the third white light image and the fourth white light image are acquired for the Nth time, determine the correction coefficient corresponding to N; wherein, the larger N is, the larger the correction coefficient is, and N can be a positive integer; the control module 73 is also used to, after determining the third exposure parameter and the fourth exposure parameter, determine the updated third exposure parameter based on the third exposure parameter and the correction coefficient; and determine the updated fourth exposure parameter based on the fourth exposure parameter and the correction coefficient.

[0197] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Figure 8 As shown, the electronic device includes: a processor 81 and a machine-readable storage medium 82, wherein the machine-readable storage medium 82 stores machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image processing method of the 3D endoscope disclosed in the above example of this application.

[0198] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method of the 3D endoscope disclosed in the above example of the present application can be implemented.

[0199] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0200] Based on the same application concept as the above method, the embodiment of the present application further provides a computer program product, which may include a computer program. When the computer program is executed by a processor, the image processing method of the 3D endoscope disclosed in the above example of the present application is implemented.

[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0202] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A 3D endoscope image processing method, characterized in that: If the 3D endoscope adopts a white fluorescence stroboscopic mode, white light and fluorescence are alternately outputted by a light source; when the light source outputs white light, a first white light image is collected by a first sensor, and a second white light image is collected by a second sensor; when the light source outputs fluorescence, a first fluorescence image is collected by the first sensor, and a second fluorescence image is collected by the second sensor. The method includes: Acquire a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image; Determine a first average brightness of a first current frame image and a second average brightness of a second current frame image, wherein the first average brightness is determined based on a brightness value of each pixel of the first current frame image; Determine a target change multiple based on the first average brightness and a configured target brightness, wherein the target change multiple represents an expected change ratio when changing from the first average brightness to the target brightness; Determine a first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image, wherein the first exposure parameter includes at least one of light source intensity, sensor gain, and a first algorithm digital gain; determine a second exposure parameter based on the first average brightness, the second average brightness, and the first algorithm digital gain, wherein the second exposure parameter includes a second algorithm digital gain; Among them, if the first current frame image is a first white light image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame white light image; if the first current frame image is a first fluorescent image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame fluorescent image.

2. The method according to claim 1, characterized in that The determining a first average brightness of the first current frame image includes: Dividing the first current frame image into a plurality of image blocks, and determining, for each image block, a block average brightness of the image block based on a brightness value of each pixel point of the image block; Acquire a first image block located in an effective field of view area and a second image block not located in the effective field of view area, wherein the effective field of view area is an effective illumination area of ​​the light source; The first average brightness is determined based on the block average brightness of non-highlight blocks in the first image block, the block average brightness of highlighted blocks in the first image block, the acquired target overexposure weight and the total number of first image blocks; wherein the target overexposure weight represents the weight corresponding to the block average brightness of the highlighted blocks.

3. The method according to claim 2, characterized in that Before acquiring the first image block located in the effective field of view area and the second image block not located in the effective field of view area, the method further comprises: Determine a brightness threshold based on a ratio of a central area brightness of the first current frame image and a configured brightness threshold, wherein the central area brightness is an average value of brightness values ​​of each image block in the central area; Determine a first boundary and a second boundary of a first dimension, and determine a third boundary and a fourth boundary of a second dimension based on the first boundary, the second boundary, and the width and height of the first current frame image; Wherein, if the first dimension is a column dimension and the second dimension is a row dimension, the first boundary is the first column that meets the preset condition from the left, and the second boundary is the first column that meets the preset condition from the right; if the first dimension is a row dimension and the second dimension is a column dimension, the first boundary is the first row that meets the preset condition from the top, and the second boundary is the first row that meets the preset condition from the bottom; a column that meets the preset condition means that the average value of the block average brightness of all image blocks in the column is not less than the brightness threshold; a row that meets the preset condition means that the average value of the block average brightness of all image blocks in the row is not less than the brightness threshold; If the image block is located in the area formed by the first boundary, the second boundary, the third boundary and the fourth boundary, the image block is the first image block; otherwise, the image block is the second image block.

4. The method according to claim 2, characterized in that: The method for determining the target overexposure weight includes: determining the target highlight ratio based on the total number of highlight blocks and the total number of first image blocks; wherein, for each first image block, if the block average brightness of the first image block is not less than the highlight block brightness threshold, then the first image block belongs to the highlight block; if the block average brightness of the first image block is less than the highlight block brightness threshold, then the first image block does not belong to the highlight block; The target overexposure weight is determined based on the target highlight ratio; wherein, the larger the target highlight ratio is, the larger the target overexposure weight is; wherein, multiple threshold intervals are pre-configured, different threshold intervals correspond to different mapping relationships, and the mapping relationship represents the relationship between the highlight ratio and the overexposure weight; the target overexposure weight is obtained by querying the mapping relationship corresponding to the threshold interval to which the target highlight ratio belongs.

5. The method according to claim 4, characterized in that The determining the target overexposure weight based on the target highlight ratio includes: If the target highlight ratio is within the threshold interval [0, size1), the target overexposure weight is determined using the following formula: If the target highlight ratio is within the threshold interval [size1, size2), the target overexposure weight is determined using the following formula: If the target highlight ratio is within the threshold interval [size2,1], the target overexposure weight is determined using the following formula: Among them, high_light_blk_w represents the target overexposure weight, high_light_w represents the target highlight ratio, w1, w2, w3, size1 and size2 represent the configured values; The determining the first average brightness based on the block average brightness of non-highlight blocks in the first image block, the block average brightness of highlight blocks in the first image block, the acquired target overexposure weight and the total number of first image blocks includes: determining the first average brightness using the following formula: Wherein, y_ave represents the first average brightness, valid_blk_num represents the total number of first image blocks, K1 represents the sum of the block average brightness of all non-highlight blocks, and K2 represents the sum of the block average brightness of all highlight blocks.

6. The method according to claim 1, characterized in that The determining the target change multiple based on the first average brightness and the configured target brightness includes: Correcting the first average brightness based on the brightness information of the historical frame image to obtain a brightness estimation value of the first current frame image; determining a first difference between the target brightness and the brightness estimation value; Determining a target change multiple of a first current frame image based on the first difference and the second difference; wherein the second difference is a difference between the target brightness and a brightness estimation value of a historical frame image; If the first current frame image is the first white light image, the historical frame image is the historical frame white light image; if the first current frame image is the first fluorescent image, the historical frame image is the historical frame fluorescent image.

7. The method according to claim 6, characterized in that The brightness information of the historical frame image includes an average brightness and a brightness estimation value of the historical frame image. The first average brightness is corrected based on the brightness information of the historical frame image to obtain the brightness estimation value of the first current frame image, including: Determine a brightness prediction value of a first current frame image based on the average brightness of the historical frame images, and perform a weighted operation on the first average brightness and the brightness prediction value to obtain an average brightness correction value; A weighted operation is performed on the brightness estimation value of the historical frame image and the average brightness correction value to obtain a brightness estimation value of the first current frame image.

8. The method according to claim 6, characterized in that The determining the target change multiple of the first current frame image based on the first difference and the second difference includes: Determining a brightness control amount corresponding to the brightness estimation value based on the first difference, the second difference, a configured proportional adjustment coefficient, and a configured differential adjustment coefficient; An initial change multiple of the first current frame image is determined based on the brightness control amount and the brightness estimation value; a weighted operation is performed on the initial change multiple of the first current frame image and the target change multiple of the historical frame image to obtain the target change multiple of the first current frame image.

9. The method according to claim 1, characterized in that: The determining the first exposure parameter based on the target change multiple and the exposure parameter of the first current frame image includes: If the target change multiple is greater than a preset change multiple threshold, then increasing the light source intensity in the exposure parameters of the first current frame image to obtain the light source intensity in the first exposure parameters; and / or increasing the sensor gain in the exposure parameters of the first current frame image to obtain the sensor gain in the first exposure parameters; and / or increasing the first algorithm digital gain in the exposure parameters of the first current frame image to obtain the first algorithm digital gain in the first exposure parameters; If the target change multiple is less than the preset change multiple threshold, reduce the light source intensity in the exposure parameters of the first current frame image to obtain the light source intensity in the first exposure parameters; and / or reduce the sensor gain in the exposure parameters of the first current frame image to obtain the sensor gain in the first exposure parameters; and / or reduce the first algorithm digital gain in the exposure parameters of the first current frame image to obtain the first algorithm digital gain in the first exposure parameters.

10. The method according to claim 1, characterized in that Determining a second algorithm digital gain based on the first average brightness, the second average brightness, and the first algorithm digital gain includes: determining a quotient between the second average brightness and the first average brightness; If the quotient is less than a preset ratio threshold, the second algorithm digital gain is determined based on the product value between the first algorithm digital gain and the quotient; or, if the quotient is not less than the preset ratio threshold, the first algorithm digital gain is determined as the second algorithm digital gain.

11. The method according to any one of claims 1 to 10, characterized in that: If the 3D endoscope adopts the white light mode, the light source continuously outputs white light, the first sensor collects the third white light image, and the second sensor collects the fourth white light image, and the method further includes: determining a third average brightness of the third white-light image and a fourth average brightness of the fourth white-light image; Determine a target change multiple based on the third average brightness and the target brightness; determine a third exposure parameter of a next frame of white light image based on the target change multiple and an exposure parameter of a third white light image, wherein the third exposure parameter includes at least one of a shutter time, a sensor gain, and a third algorithm digital gain; A fourth exposure parameter of a next frame of white light image is determined based on the third average brightness, the fourth average brightness and the third algorithm digital gain, where the fourth exposure parameter includes a fourth algorithm digital gain.

12. The method according to claim 11, characterized in that If the 3D endoscope is switched from the white light mode to the white fluorescent stroboscopic mode, for the mode switching process, before acquiring the first current frame image and the second current frame image for the first time, the first exposure parameter is set to the first fixed exposure parameter, and the second exposure parameter is set to the second fixed exposure parameter; When the first current frame image and the second current frame image are acquired for the Mth time, a correction coefficient corresponding to M is determined; wherein the larger M is, the larger the correction coefficient is; after determining the first exposure parameter and the second exposure parameter, an updated first exposure parameter is determined based on the first exposure parameter and the correction coefficient; and an updated second exposure parameter is determined based on the second exposure parameter and the correction coefficient; Alternatively, if the 3D endoscope is switched from the white fluorescent stroboscopic mode to the white light mode, for the mode switching process, before acquiring the third white light image and the fourth white light image for the first time, the third exposure parameter is set to the third fixed exposure parameter, and the fourth exposure parameter is set to the fourth fixed exposure parameter; When the third white light image and the fourth white light image are acquired for the Nth time, the correction coefficient corresponding to N is determined; wherein, the larger N is, the larger the correction coefficient is; after determining the third exposure parameter and the fourth exposure parameter, an updated third exposure parameter is determined based on the third exposure parameter and the correction coefficient; and an updated fourth exposure parameter is determined based on the fourth exposure parameter and the correction coefficient.

13. An image processing device for a 3D endoscope, characterized in that: If the 3D endoscope adopts the white fluorescence stroboscopic mode, the light source alternately outputs white light and fluorescence; when the light source outputs white light, the first white light image is collected by the first sensor, and the second white light image is collected by the second sensor; when the light source outputs fluorescence, the first fluorescence image is collected by the first sensor, and the second fluorescence image is collected by the second sensor. The device includes: An acquisition module, used to acquire a first current frame image and a second current frame image; wherein, if the first current frame image is a first white light image, the second current frame image is a second white light image, or, if the first current frame image is a first fluorescent image, the second current frame image is a second fluorescent image; A determination module, configured to determine a first average brightness of a first current frame image and a second average brightness of a second current frame image, wherein the first average brightness is determined based on the brightness value of each pixel of the first current frame image; and determine a target change multiple based on the first average brightness and a configured target brightness, wherein the target change multiple represents an expected change ratio when the first average brightness changes to the target brightness; a control module, configured to determine a first exposure parameter based on the target change multiple and an exposure parameter of a first current frame image, wherein the first exposure parameter includes at least one of light source intensity, sensor gain, and a first algorithm digital gain; and determine a second exposure parameter based on the first average brightness, the second average brightness, and the first algorithm digital gain, wherein the second exposure parameter includes a second algorithm digital gain; Among them, if the first current frame image is a first white light image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame white light image; if the first current frame image is a first fluorescent image, the first exposure parameter and the second exposure parameter are exposure parameters of the next frame fluorescent image.

14. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-12.

Images