Endoscope imaging method and device, electronic equipment and storage medium

Abstract

The invention discloses an endoscope imaging method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first white light image and a first fluorescence image of an object through a first image sensor, and obtaining a second white light image and a second fluorescence image of the object through a second image sensor; determining depth information corresponding to fluorescent pixel points in the first fluorescent image and / or the second fluorescent image according to the first white light image and the second white light image; determining a reference depth, and determining reference fluorescence receiving intensity corresponding to the reference depth; determining fluorescence receiving intensity variation corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth and the reference fluorescence receiving intensity; and correcting the fluorescence receiving intensity of the fluorescence pixel points according to the fluorescence receiving intensity variation to obtain a corrected first fluorescence image and / or a corrected second fluorescence image. The negative influence of the change degree of the fluorescence signal along with the depth on the display effect of the fluorescence image is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of medical devices, and more particularly, to an endoscope imaging method, apparatus, electronic device, and storage medium. Background Art

[0002] Fluorescence endoscopes collect fluorescence signals in the body by virtue of the property that special fluorescent dyes emit fluorescence signals under illumination of a specific wavelength, and perform processing, analysis, and display through image processing techniques. The main advantage of fluorescence endoscopes is that they can enhance the contrast between regions of interest (such as lesion regions) and normal tissues (unstained regions). Usually, fluorescent dyes emit bright fluorescence signals after excitation, forming a sharp contrast with the surrounding tissues. However, when collecting fluorescence signals in real time using a fluorescence endoscope, the acquisition distance, i.e., the depth, changes with the needs of the surgery or observation. In the case where the concentration of the fluorescent dye in the region remains unchanged, the fluorescence signal still changes with the distance, and the display effect of the fluorescence image affects the observation and diagnosis of the surgery.

[0003] Therefore, how to reduce the negative impact of the change of the fluorescence signal with depth on the display effect of the fluorescence image is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present application is to provide an endoscope imaging method, apparatus, an electronic device, and a computer-readable storage medium, which reduce the negative impact of the degree of change of the fluorescence signal with depth on the display effect of the fluorescence image.

[0005] To achieve the above purpose, the present application provides an endoscope imaging method, including:

[0006] Obtaining a first white light image and a first fluorescence image of an object through a first image sensor, and obtaining a second white light image and a second fluorescence image of the object through a second image sensor;

[0007] Determining depth information corresponding to fluorescent pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image;

[0008] Determining a reference depth and determining a reference fluorescence reception intensity corresponding to the reference depth;

[0009] Determining a change amount of the fluorescence reception intensity corresponding to the fluorescent pixel points according to the depth information corresponding to the fluorescent pixel points, the reference depth, and the reference fluorescence reception intensity;

[0010] Correcting the fluorescence reception intensity of the fluorescent pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0011] Among them, determining the depth information corresponding to the fluorescent pixel points in the first fluorescent image and the second fluorescent image according to the first white light image and the second white light image includes:

[0012] Determine the depth information of the white light pixel points according to the first white light image and the second white light image, and use it as the depth information corresponding to the fluorescent pixel points in the first fluorescent image and / or the second fluorescent image.

[0013] Among them, before determining the depth information corresponding to the fluorescent pixel points in the first fluorescent image and the second fluorescent image according to the first white light image and the second white light image, it further includes:

[0014] Interpolate the first fluorescent image and / or the second fluorescent image to make its resolution the same as that of the first white light image and the second white light image.

[0015] Among them, determining the depth information of the white light pixel points according to the first white light image and the second white light image includes:

[0016] Determine the first difference in the target direction of the coordinates of the corresponding white light pixel points of the same object point in the first white light image and the second white light image; wherein, the target direction is the direction of the line connecting the optical centers of the first camera and the second camera, the first camera is the camera corresponding to the first image sensor, and the second camera is the camera corresponding to the second image sensor;

[0017] Determine the first product of the distance between the optical center of the first camera and the optical center of the second camera and the target focal length; wherein, the target focal length is the focal length of the first camera and the second camera;

[0018] Determine the first ratio of the first product to the first difference as the depth information corresponding to the white light pixel points corresponding to the object point in the first white light image and the second white light image.

[0019] Among them, determining the reference depth and determining the reference fluorescence reception intensity corresponding to the reference depth includes:

[0020] Determine the minimum depth information and the maximum depth information of the white light pixel points, and determine the reference depth between the minimum depth information and the maximum depth information;

[0021] Determine the reference fluorescence reception intensity corresponding to the reference depth according to the first fluorescent image or the second fluorescent image.

[0022] Among them, determining the fluorescence reception intensity variation corresponding to the fluorescence pixel based on the depth information corresponding to the fluorescence pixel, the reference depth, and the reference fluorescence reception intensity includes:

[0023] Determining a second ratio between the square of the reference depth and the square of the depth information corresponding to the fluorescence pixel, and determining a third ratio between the second ratio and a preset adjustment factor;

[0024] Determining a second product between the third ratio and the reference fluorescence reception intensity, and determining a second difference between the reference fluorescence reception intensity and the second product as the fluorescence reception intensity variation corresponding to the fluorescence pixel;

[0025] Correspondingly, correcting the fluorescence reception intensity of the fluorescence pixel according to the fluorescence reception intensity variation to obtain a corrected first fluorescence image and / or a corrected second fluorescence image includes:

[0026] Correcting the fluorescence reception intensity of the fluorescence pixel to the sum of the fluorescence reception intensity of the fluorescence pixel and the fluorescence reception intensity variation to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0027] Among them, the method further includes:

[0028] Fusing the first white light image and the corrected first fluorescence image to obtain a first fused image, and fusing the second white light image and the corrected second fluorescence image to obtain a second fused image;

[0029] Determining a target image according to the first fused image and the second fused image, and outputting the target image.

[0030] Among them, determining the target image according to the first fused image and the second fused image includes:

[0031] Determining the first fused image or the second fused image as the target image.

[0032] Among them, determining the target image according to the first fused image and the second fused image includes:

[0033] Generating a stereoscopic image as the target image according to the first fused image and the second fused image.

[0034] Among them, after obtaining the first white light image and the first fluorescence image of the object through the first image sensor, and obtaining the second white light image and the second fluorescence image of the object through the second image sensor, it further includes:

[0035] Perform demosaicing operations on the first white light image and the second white light image.

[0036] To achieve the above object, the present application provides an endoscope imaging device, including:

[0037] An acquisition unit, configured to acquire a first white light image and a first fluorescence image of an object through a first image sensor, and acquire a second white light image and a second fluorescence image of the object through a second image sensor;

[0038] A first determination unit, configured to determine depth information corresponding to fluorescence pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image;

[0039] A second determination unit, configured to determine a reference depth and determine a reference fluorescence reception intensity corresponding to the reference depth;

[0040] A third determination unit, configured to determine a change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity;

[0041] A correction unit, configured to correct the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0042] To achieve the above object, the present application provides an electronic device, including:

[0043] A memory, configured to store a computer program;

[0044] A processor, configured to implement the steps of the above endoscope imaging method when executing the computer program.

[0045] To achieve the above object, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above endoscope imaging method are implemented.

[0046] As can be seen from the above solution, an endoscope imaging method provided by the present application includes: acquiring a first white light image and a first fluorescence image of an object through a first image sensor, and acquiring a second white light image and a second fluorescence image of the object through a second image sensor; determining depth information corresponding to fluorescence pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image; determining a reference depth and determining a reference fluorescence reception intensity corresponding to the reference depth; determining a change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity; and correcting the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0047] In the present application, endoscope imaging and correction of fluorescence images can be achieved only through two image sensors, namely a first image sensor and a second image sensor, which reduces the bandwidth requirement for data transmission and the power consumption of the endoscope imaging system, and can be applied to rigid endoscopes or flexible endoscopes. At the same time, in the endoscope imaging method provided by the present application, the fluorescence reception intensity is corrected according to the depth information corresponding to the pixel points in the fluorescence image, strengthening the fluorescence of the pixel points with a greater depth, that is, a greater distance, and weakening the fluorescence of the pixel points with a shallower depth, that is, a shorter distance, thereby reducing the negative impact of the change of the fluorescence signal with depth on the display effect of the fluorescence image. The corrected fluorescence image (and other images generated therefrom, such as white light-fluorescence fusion images, three-dimensional white light-fluorescence fusion images, three-dimensional fluorescence images, etc.) can better display the fluorescence region, especially strengthening the display of the edge of the region of interest, which is beneficial to the diagnosis and treatment of the affected part. The present application also discloses an endoscope imaging device, an electronic device, and a computer-readable storage medium, which can also achieve the above technical effects.

[0048] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification, and are used together with the following specific embodiments to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0050] Figure 1Structural diagram of an endoscopic imaging system shown according to an exemplary embodiment;

[0051] Figure 2 Flowchart of an endoscopic imaging method shown according to an exemplary embodiment;

[0052] Figure 3 Schematic diagram of calculating depth information shown according to an exemplary embodiment;

[0053] Figure 4 Schematic diagram of the relationship between fluorescence signal intensity and depth shown according to an exemplary embodiment;

[0054] Figure 5 Flowchart of another endoscopic imaging method shown according to an exemplary embodiment;

[0055] Figure 6 Structural diagram of an endoscopic imaging device shown according to an exemplary embodiment;

[0056] Figure 7 Structural diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence.

[0058] To facilitate understanding of the endoscopic imaging method provided in the present application, first, the endoscopic imaging system applied in the present application will be introduced, as Figure 1 shown, including a light source module, an imaging module, an image processing module, a microprocessor module, etc.

[0059] The light source module includes a light source, illumination optics, an imaging fiber bundle, an illumination lens, etc. The light source can be an LED (Light Emitting Diode) light source, which is a multi-wavelength visible light source and a fluorescence excitation light source. The illumination optics collects the light from the illumination source and focuses it on the imaging fiber bundle; the end face of the imaging fiber bundle is located on the object plane of the illumination lens.

[0060] The imaging module includes two image sensors, as well as corresponding drive circuits, signal lines, a camera interface, and an imaging lens. The two image sensors are respectively the image sensors corresponding to the left and right eyes, and can be RGBIR image sensors. The RGB pixels are responsible for receiving white light, and the IR pixels are responsible for receiving fluorescence. That is, the RGBIR image sensors are used to simultaneously collect white light images and fluorescence images. The image sensors are located on the image plane of the imaging lens, and the signal lines connect the drive circuits and the camera interface. The imaging module transmits the collected image information to the image processing module.

[0061] The image processing module includes a series of algorithms from sensor image data to the image output on the display. The image processing module is used to obtain the depth information of the object to the image sensor using the binocular measurement principle, correct the fluorescence signal in combination with the depth information to reduce the degree of signal variation with distance, and fuse the white light image and the fluorescence image. It can be the fusion of a single-channel white light image and fluorescence image, that is, fuse the white light image and the fluorescence image corresponding to one of the image sensors. It can also be the fusion of a two-channel white light image and fluorescence image, that is, fuse the images after fusing the white light images and fluorescence images corresponding to the two image sensors again to form an image with a stereoscopic display effect.

[0062] The microprocessor module is used to control the camera, etc.

[0063] The applicant found that in the related art, the structures of 3D fluorescence endoscopes are relatively complex. For example, two groups of image sensors for the left and right eyes are required, and each group corresponds to a white light sensor and a fluorescence sensor. That is, a total of four sensors are required to achieve endoscope imaging. Too many sensors have high requirements for the data transmission bandwidth, high power consumption, difficult heat dissipation, and the structural size of the endoscope is too large, and it can only be adapted to optical mirrors. The endoscope imaging system provided in this application can achieve endoscope imaging only using two image sensors, and can be applied to rigid endoscopes or flexible endoscopes.

[0064] The embodiment of this application discloses an endoscope imaging method, which reduces the negative impact of the degree of fluorescence signal variation with depth on the display effect of the fluorescence image.

[0065] See Figure 2 , according to the flowchart of an endoscope imaging method shown in an exemplary embodiment, as Figure 2 shown, including:

[0066] S101: Obtain a first white light image and a first fluorescence image of the object through a first image sensor, and obtain a second white light image and a second fluorescence image of the object through a second image sensor;

[0067] The execution subject of this embodiment is the image processing module in the above endoscopic imaging system. In a specific implementation, a first white light image and a first fluorescence image of an object are acquired by a first image sensor, and a second white light image and a second fluorescence image of the object are acquired by a second image sensor. The first image sensor and the second image sensor in this embodiment are respectively the image sensors corresponding to the left and right eyes, and the first image sensor and the second image sensor can be RGBIR image sensors for simultaneously collecting white light images and fluorescence images. The resolutions of the first image sensor and the second image sensor are the same, that is, the resolution of the first white light image collected by the first image sensor is the same as that of the second white light image collected by the second image sensor, and the resolution of the first fluorescence image collected by the first image sensor is the same as that of the second fluorescence image collected by the second image sensor.

[0068] As a preferred implementation manner, after acquiring the first white light image and the first fluorescence image of the object by the first image sensor and acquiring the second white light image and the second fluorescence image of the object by the second image sensor, it further includes: performing a demosaicing operation on the first white light image and the second white light image. It can be understood that since the RGBIR image sensor belongs to the Bayer template in the RGBIR form, it is necessary to perform a demosaicing operation on the single-channel signal to obtain a three-channel RGB signal, and the resolutions of the three-channel RGB signals are consistent.

[0069] S102: Determine the depth information corresponding to the fluorescent pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image;

[0070] If an output image needs to be obtained according to the first fluorescence image in the subsequent process, then in this step, determine the depth information corresponding to the fluorescent pixel points in the first fluorescence image according to the first white light image and the second white light image. If an output image needs to be obtained according to the second fluorescence image in the subsequent process, then in this step, determine the depth information corresponding to the fluorescent pixel points in the second fluorescence image according to the first white light image and the second white light image. If an output image needs to be obtained according to both the first fluorescence image and the second fluorescence image in the subsequent process, then in this step, determine the depth information corresponding to the fluorescent pixel points in both the first fluorescence image and the second fluorescence image according to the first white light image and the second white light image simultaneously.

[0071] As a feasible implementation manner, the determining the depth information corresponding to the fluorescent pixel points in the first fluorescence image and the second fluorescence image according to the first white light image and the second white light image includes: determining the depth information of the white light pixel points according to the first white light image and the second white light image, and using it as the depth information corresponding to the fluorescent pixel points in the first fluorescence image and / or the second fluorescence image.

[0072] In a specific implementation, a binocular vision measurement method is used to determine the depth information corresponding to the fluorescent pixel points in the fluorescent image. It can be understood that since the white light image is clearer and has richer details than the fluorescent image, the binocular vision measurement method can be used to determine the depth of each white light pixel point in the first white light image and the second white light image, and then correspond them one by one to the fluorescent pixel points in the fluorescent image.

[0073] As a preferred implementation, before determining the depth information corresponding to the fluorescent pixel points in the first fluorescent image and the second fluorescent image according to the first white light image and the second white light image, it further includes: interpolating the first fluorescent image and / or the second fluorescent image to make its resolution the same as that of the first white light image and the second white light image.

[0074] It can be understood that since the resolution of the IR native signal is different from that of the three-channel RGB signal, it is necessary to interpolate the IR native signal into the same resolution as the above three-channel RGB signal, that is, to interpolate the first fluorescent image and the second fluorescent image into the same resolution as the first white light image and the second white light image, so as to correspond the depth information of the pixels in the white light image to the pixels in the fluorescent image in the subsequent steps.

[0075] As a feasible implementation, determining the depth information of the white light pixel points according to the first white light image and the second white light image includes: determining the first difference in the target direction of the coordinates of the white light pixel points corresponding to the same object point in the first white light image and the second white light image; wherein, the target direction is the direction of the line connecting the optical centers of the first camera and the second camera, the first camera is the camera corresponding to the first image sensor, and the second camera is the camera corresponding to the second image sensor; determining the first product of the distance between the optical centers of the first camera and the second camera and the target focal length; wherein, the target focal length is the focal length of the first camera and the second camera; determining the first ratio of the first product to the first difference as the depth information corresponding to the white light pixel points corresponding to the object point in the first white light image and the second white light image.

[0076] In a specific implementation, as Figure 3 shown, point P is the object point, and O l is the optical center of the first camera (i.e., the left camera), and O r is the optical center of the second camera (i.e., the right camera), f is the focal length of the first camera and the second camera, x l is the coordinate value of the white light pixel point corresponding to point P in the first white light image in the target direction, and x ris the coordinate value of the white light pixel corresponding to point P in the second white light image in the target direction, where the target direction is the direction of the line connecting the optical centers of the first camera and the second camera, and Δδ is the difference in the coordinates of the white light pixels corresponding to point P in the first white light image and the second white light image in the target direction, that is, x l -x r . D is the depth information of the white light pixels corresponding to point P in the first white light image and the second white light image. The calculation formula for the depth information D of point P is:

[0077]

[0078] Since the white light signal and the fluorescence signal come from the same image sensor and the resolutions have been processed to be the same, the depth information of the white light pixels corresponding to point P in the first white light image and the second white light image is also the depth information of the fluorescence pixels corresponding to point P in the first fluorescence image and the second fluorescence image.

[0079] S103: Determine the reference depth and determine the reference fluorescence reception intensity corresponding to the reference depth;

[0080] In this step, determine the reference depth and the reference fluorescence reception intensity corresponding to it, so that in the subsequent steps, the fluorescence reception intensity can be corrected according to the reference depth and the reference fluorescence reception intensity.

[0081] As a feasible implementation manner, the determining the reference depth and determining the reference fluorescence reception intensity corresponding to the reference depth includes: determining the minimum depth information and the maximum depth information of the white light pixels, and determining the reference depth between the minimum depth information and the maximum depth information; determining the reference fluorescence reception intensity corresponding to the reference depth according to the first fluorescence image or the second fluorescence image. In a specific implementation, determine the reference depth between the minimum depth information and the maximum depth information of the white light pixels, and determine the reference fluorescence reception intensity corresponding to this reference depth in the first fluorescence image or the second fluorescence image. For example, the average value of the minimum depth information and the maximum depth information can be selected from the depth information of all white light pixels as the reference depth, or the reference depth can be determined by other calculation methods, and the reference fluorescence reception intensity is determined according to the reference depth, which can be used to correct the fluorescence image. Therefore, the setting scheme of the reference value can be adjusted according to the requirements of the display effect of the image. For example, if you want to enhance the edge of the fluorescence image, the value with the shallowest (nearest) depth and the largest fluorescence reception intensity can be used as the reference value to enhance the pixels with poor display effects in the fluorescence image.

[0082] S104: Determine the change amount of the fluorescence reception intensity corresponding to the fluorescence pixel according to the depth information corresponding to the fluorescence pixel, the reference depth, and the reference fluorescence reception intensity;

[0083] S105: Correct the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0084] In a specific implementation, the fluorescence reception intensity is corrected according to the depth information corresponding to the pixel points in the fluorescence image, the reference depth, and the reference fluorescence reception intensity. Increase the fluorescence reception intensity of the fluorescence pixel points with depth information greater than the reference depth, and decrease the fluorescence reception intensity of the fluorescence pixel points with depth information less than the reference depth, so as to reduce the degree of change of the fluorescence signal with depth, and obtain a corrected first fluorescence image and a corrected second fluorescence image.

[0085] As a feasible implementation manner, determining the change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity includes: determining a second ratio between the square of the reference depth and the square of the depth information corresponding to the fluorescence pixel points, and determining a third ratio between the second ratio and a preset adjustment factor; determining a second product between the third ratio and the reference fluorescence reception intensity, and determining the second difference between the reference fluorescence reception intensity and the second product as the change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points; correspondingly, correcting the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image includes: correcting the fluorescence reception intensity of the fluorescence pixel points to the sum of the fluorescence reception intensity of the fluorescence pixel points and the change amount of the fluorescence reception intensity, so as to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0086] It should be noted that according to the law of inverse square of illuminance and distance, the illuminance on the detection surface is inversely proportional to the square of the distance between the light source and the detection surface, that is, the fluorescence signal intensity I is inversely proportional to the depth D. As Figure 4 shown, the fluorescence reception intensity corresponding to the distance d1 is I1, and the fluorescence reception intensity corresponding to the distance d2 is I2, and the relational expression can be obtained:

[0087]

[0088] where s is an adjustment factor, which can be 1. It can be deduced that when the distance is adjusted from d1 to d2, the change amount of the fluorescence reception intensity is:

[0089]

[0090] In order to make the fluorescence intensity invariant with distance, the correction amount of the fluorescence reception intensity should be equal to the change amount of the fluorescence reception intensity. In a specific implementation, after enabling the fluorescence correction function, a reference depth is selected, and the reference fluorescence reception intensity corresponding to the reference depth is determined. For example, the above d1 is selected as the reference depth and I1 as the reference fluorescence reception intensity. For a fluorescence pixel point with a depth information of d2, the actually measured fluorescence reception intensity is I2. Then, the change amount of the fluorescence reception intensity, that is, the correction amount of the fluorescence reception intensity, is calculated according to the calculation formula of the change amount of the fluorescence reception intensity. The fluorescence reception intensity of the fluorescence pixel point is corrected to: I = I2 + ΔI, where I is the corrected fluorescence reception intensity.

[0091] It should be noted that in this embodiment, the corrected first fluorescence image or the corrected second fluorescence image can be selected for direct output, and the output image is a two-dimensional fluorescence image. Alternatively, after fusing the corrected first fluorescence image or the corrected second fluorescence image, the output image is a three-dimensional fluorescence image, which is a three-dimensional image and has a stereoscopic display effect after passing through a 3D monitor.

[0092] On this basis, as a preferred implementation manner, this embodiment further includes: fusing the first white light image and the corrected first fluorescence image to obtain a first fused image, and fusing the second white light image and the corrected second fluorescence image to obtain a second fused image; determining a target image according to the first fused image and the second fused image, and outputting the target image.

[0093] In a specific implementation, the first white light image and the corrected first fluorescence image are fused to obtain a first fused image, and the second white light image and the corrected second fluorescence image are fused to obtain a second fused image. The specific fusion method is not limited here. The target image to be output is determined according to the first fused image and the second fused image.

[0094] As a feasible implementation manner, determining the target image according to the first fused image and the second fused image includes: determining the first fused image or the second fused image as the target image. In a specific implementation, the first fused image or the second fused image can be selected for direct output, that is, a white light-fluorescence fused image is output, which is a two-dimensional image and has no stereoscopic display effect.

[0095] As another feasible implementation manner, determining the target image according to the first fused image and the second fused image includes: generating a stereoscopic image as the target image according to the first fused image and the second fused image. In specific implementation, the first fused image and the second fused image can be fused to obtain the target image for output, and the specific fusion manner is not limited herein. For example, the interlaced fusion, up-down fusion, left-right fusion, etc. of stereoscopic display can be adopted, and the output target image is a stereoscopic white light-fluorescence fused image, which is a three-dimensional image and has a stereoscopic display effect after passing through a 3D display.

[0096] For an RGBIR image sensor, since it simultaneously acquires a white light image and a fluorescence image, in the related art, it is impossible to separately adjust the white light signal and the fluorescence signal when adjusting parameters such as the exposure time and the gain that affect the signal strength. However, due to the adoption of the fluorescence correction scheme of this embodiment, when adjusting the parameters of the image sensor, the white light signal is adjusted, and at the same time, the fluorescence signal is corrected, so that the fluorescence signal remains unchanged or changes very little, that is, the white light signal is separately adjusted. When it is necessary to separately adjust the fluorescence signal, the parameters of the image sensor can be finely adjusted so that the signal change of the white light signal is very small, and at the same time, the above adjustment factor s is increased so that the correction degree of the fluorescence signal is greater, realizing the adjustment of the fluorescence signal. It can be seen that through the fluorescence correction scheme of this embodiment, the separate adjustment of the white light signal and the fluorescence signal is realized.

[0097] In the embodiment of the present application, only two image sensors, namely the first image sensor and the second image sensor, can be used to realize the correction of the endoscope imaging and the fluorescence image, reducing the bandwidth requirement for data transmission and the power consumption of the endoscope imaging system, and can be applicable to rigid endoscopes or flexible endoscopes. At the same time, the endoscope imaging method provided by the embodiment of the present application corrects the fluorescence reception intensity according to the depth information corresponding to the pixel points in the fluorescence image, strengthens the fluorescence of the pixel points with a greater depth, that is, a farther distance, and weakens the fluorescence with a shallower depth, that is, a closer distance, thereby reducing the negative impact of the change of the fluorescence signal with the depth on the display effect of the fluorescence image. The corrected fluorescence image (and other images generated thereby, such as the white light-fluorescence fused image, the stereoscopic white light-fluorescence fused image, the stereoscopic fluorescence image, etc.) can better display the fluorescence region, especially strengthen the display of the edge of the region of interest, which is beneficial to the diagnosis and treatment of the affected part.

[0098] The embodiment of the present application discloses an endoscope imaging method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0099] See Figure 5 , according to the flowchart of another endoscope imaging method shown in an exemplary embodiment, as Figure 5 shown, includes:

[0100] S201: Obtain a first white light image and a first fluorescence image of an object through a first image sensor, and obtain a second white light image and a second fluorescence image of the object through a second image sensor;

[0101] S202: Determine the depth information corresponding to the fluorescent pixel points in the first fluorescence image and the second fluorescence image according to the first white light image and the second white light image;

[0102] S203: Determine a reference depth and determine the reference fluorescence reception intensity corresponding to the reference depth;

[0103] S204: Determine the change amount of the fluorescence reception intensity corresponding to the fluorescent pixel points according to the depth information corresponding to the fluorescent pixel points, the reference depth, and the reference fluorescence reception intensity;

[0104] S205: Correct the fluorescence reception intensity of the fluorescent pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and a corrected second fluorescence image;

[0105] S206: Fuse the first white light image and the corrected first fluorescence image to obtain a first fused image, and fuse the second white light image and the corrected second fluorescence image to obtain a second fused image;

[0106] S207: Determine whether a stereoscopic display image needs to be output; if yes, enter S208; if no, enter S209;

[0107] S208: Generate a stereoscopic image as a target image according to the first fused image and the second fused image, and output the target image;

[0108] S209: Output the first fused image or the second fused image.

[0109] In this embodiment, when a stereoscopic display image does not need to be output, the first fused image or the second fused image is selected and directly output, and the output image is a two-dimensional image without a stereoscopic display effect. When a stereoscopic display image needs to be output, the first fused image and the second fused image are fused to obtain a target image for output, and the output target image is a three-dimensional image, which has a stereoscopic display effect after passing through a 3D display.

[0110] It can be seen that this embodiment can select to output a two-dimensional image or a three-dimensional image according to requirements.

[0111] Next, an endoscope imaging device provided by an embodiment of the present application will be introduced. The endoscope imaging device described below can be referred to each other with the endoscope imaging method described above.

[0112] See Figure 6 , a structural diagram of an endoscopic imaging device shown according to an exemplary embodiment, as Figure 6 shown, includes:

[0113] An acquisition unit 601, configured to acquire a first white light image and a first fluorescence image of an object through a first image sensor, and acquire a second white light image and a second fluorescence image of the object through a second image sensor;

[0114] A first determination unit 602, configured to determine depth information corresponding to fluorescence pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image;

[0115] A second determination unit 603, configured to determine a reference depth and determine a reference fluorescence reception intensity corresponding to the reference depth;

[0116] A third determination unit 604, configured to determine a change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity;

[0117] A correction unit 605, configured to correct the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity, to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0118] In the embodiments of the present application, endoscopic imaging and correction of fluorescence images can be realized only by two image sensors, namely a first image sensor and a second image sensor, which reduces the bandwidth requirement for data transmission and reduces the power consumption of the endoscopic imaging system, and can be applicable to rigid endoscopes or flexible endoscopes. At the same time, for the endoscopic imaging device provided in the embodiments of the present application, the fluorescence reception intensity is corrected according to the depth information corresponding to the pixel points in the fluorescence image, strengthening the fluorescence of the pixel points with a greater depth, that is, a farther distance, and weakening the fluorescence with a shallower depth, that is, a closer distance, thereby reducing the negative impact of the change of the fluorescence signal with depth on the display effect of the fluorescence image. The corrected fluorescence image (and other images generated therefrom, such as a white light-fluorescence fusion image, a stereoscopic white light-fluorescence fusion image, a stereoscopic fluorescence image, etc.) can better display the fluorescence region, especially strengthening the display of the edge of the region of interest, which is beneficial to the diagnosis and treatment of the affected part.

[0119] On the basis of the above embodiments, as a preferred implementation manner, the first determination unit 602 is specifically configured to: determine the depth information of the white light pixel points according to the first white light image and the second white light image, and use it as the depth information corresponding to the fluorescence pixel points in the first fluorescence image and / or the second fluorescence image.

[0120] Based on the above embodiments, as a preferred embodiment, it further includes:

[0121] An interpolation unit for interpolating the first fluorescence image and / or the second fluorescence image to make its resolution the same as that of the first white light image and the second white light image.

[0122] Based on the above embodiments, as a preferred embodiment, the first determination unit 602 is specifically configured to: determine a first difference in the target direction of the coordinates of the white light pixel points corresponding to the same object point in the first white light image and the second white light image; wherein, the target direction is the direction of the line connecting the optical centers of the first camera and the second camera, the first camera is the camera corresponding to the first image sensor, and the second camera is the camera corresponding to the second image sensor; determine a first product of the distance between the optical center of the first camera and the optical center of the second camera and the target focal length; wherein, the target focal length is the focal length of the first camera and the second camera; determine a first ratio of the first product to the first difference as the depth information corresponding to the white light pixel points corresponding to the object point in the first white light image and the second white light image.

[0123] Based on the above embodiments, as a preferred embodiment, the second determination unit 603 is specifically configured to: determine the minimum depth information and the maximum depth information of the white light pixel points, and determine a reference depth between the minimum depth information and the maximum depth information; determine a reference fluorescence reception intensity corresponding to the reference depth according to the first fluorescence image or the second fluorescence image.

[0124] Based on the above embodiments, as a preferred embodiment, the third determination unit 604 is specifically configured to: determine a second ratio between the square of the reference depth and the square of the depth information corresponding to the fluorescence pixel point, and determine a third ratio between the second ratio and a preset adjustment factor; determine a second product of the third ratio and the reference fluorescence reception intensity, and determine a second difference between the reference fluorescence reception intensity and the second product as the change amount of the fluorescence reception intensity corresponding to the fluorescence pixel point;

[0125] Correspondingly, the correction unit 605 is specifically configured to: correct the fluorescence reception intensity of the fluorescence pixel point to the sum of the fluorescence reception intensity of the fluorescence pixel point and the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

[0126] Based on the above embodiments, as a preferred embodiment, it further includes:

[0127] A fusion unit, configured to fuse the first white light image and the corrected first fluorescence image to obtain a first fused image, and fuse the second white light image and the corrected second fluorescence image to obtain a second fused image;

[0128] An output unit, configured to determine a target image according to the first fused image and the second fused image, and output the target image.

[0129] Based on the above embodiments, as a preferred embodiment, the output unit is specifically configured to: determine the first fused image or the second fused image as the target image.

[0130] Based on the above embodiments, as a preferred embodiment, the output unit is specifically configured to: generate a stereoscopic image as the target image according to the first fused image and the second fused image.

[0131] Based on the above embodiments, as a preferred embodiment, further includes:

[0132] A demosaicing unit, configured to perform demosaicing operations on the first white light image and the second white light image.

[0133] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0134] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application further provide an electronic device, Figure 7 Shown is a structural diagram of an electronic device according to an exemplary embodiment, as Figure 7 shown, the electronic device includes:

[0135] A communication interface 1, capable of interacting with other devices such as network devices for information;

[0136] A processor 2, connected to the communication interface 1 to achieve information interaction with other devices, and when running a computer program, executes the endoscopic imaging method provided by the above one or more technical solutions. And the computer program is stored on a memory 3.

[0137] Of course, in actual application, each component in the electronic device is coupled together through a bus system 4. It can be understood that the bus system 4 is used to achieve connection and communication between these components. The bus system 4 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 7 all kinds of buses are labeled as the bus system 4.

[0138] The memory 3 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on the electronic device.

[0139] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), a ferromagnetic random access memory (FRAM, ferromagnetic random access memory), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM, Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as a static random access memory (SRAM, Static Random Access Memory), a synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory), a dynamic random access memory (DRAM, Dynamic Random Access Memory), a synchronous dynamic random access memory (SDRAM, Synchronous Dynamic Random Access Memory), a double data rate synchronous dynamic random access memory (DDR SDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), an enhanced synchronous dynamic random access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), a sync link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and a direct rambus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 3 described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memories.

[0140] The method disclosed in the embodiments of the present application above can be applied to the processor 2 or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 2 or the instructions in the form of software. The above-mentioned processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiments of the present application, it can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and combines its hardware to complete the steps of the foregoing method.

[0141] When the processor 2 executes the program, it implements the corresponding processes in each method of the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0142] In an exemplary embodiment, the embodiments of the present application also provide a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 including a stored computer program. The above computer program can be executed by the processor 2 to complete the steps described in the foregoing method. The computer-readable storage medium may be a FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, CD-ROM, or other memories.

[0143] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: various media such as removable storage devices, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0144] Alternatively, if the above integrated units of the present application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an electronic device (which can be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

[0145] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. An endoscopic imaging method, characterized in that, Including: Obtaining a first white light image and a first fluorescence image of an object through a first image sensor, and obtaining a second white light image and a second fluorescence image of the object through a second image sensor; Determining depth information corresponding to fluorescence pixel points in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image; Determining a reference depth and determining a reference fluorescence reception intensity corresponding to the reference depth; Determining a change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity; Correcting the fluorescence reception intensity of the fluorescence pixel points according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

2. The endoscopic imaging method according to claim 1, wherein The determining the depth information corresponding to the fluorescence pixel points in the first fluorescence image and the second fluorescence image according to the first white light image and the second white light image includes: Determining the depth information of white light pixel points according to the first white light image and the second white light image, and using it as the depth information corresponding to the fluorescence pixel points in the first fluorescence image and / or the second fluorescence image.

3. The endoscopic imaging method according to claim 2, wherein Before determining the depth information corresponding to the fluorescence pixel points in the first fluorescence image and the second fluorescence image according to the first white light image and the second white light image, it further includes: Interpolating the first fluorescence image and / or the second fluorescence image to make its resolution the same as that of the first white light image and the second white light image.

4. The endoscopic imaging method according to claim 2, wherein The determining the depth information of the white light pixel points according to the first white light image and the second white light image includes: Determining a first difference in the target direction of the coordinates of the white light pixel points corresponding to the same object point in the first white light image and the second white light image; wherein, the target direction is the direction of the line connecting the optical centers of the first camera and the second camera, the first camera is the camera corresponding to the first image sensor, and the second camera is the camera corresponding to the second image sensor; Determining a first product of the distance between the optical centers of the first camera and the second camera and the target focal length; wherein, the target focal length is the focal length of the first camera and the second camera; Determining a first ratio of the first product to the first difference as the depth information corresponding to the white light pixel points corresponding to the object point in the first white light image and the second white light image.

5. The endoscopic imaging method according to claim 2, characterized in that, The determining the reference depth and determining the reference fluorescence reception intensity corresponding to the reference depth includes: Determining the minimum depth information and the maximum depth information of the white light pixel points, and determining a reference depth between the minimum depth information and the maximum depth information; Determining the reference fluorescence reception intensity corresponding to the reference depth according to the first fluorescence image or the second fluorescence image.

6. The endoscopic imaging method according to claim 1, wherein The determining the change amount of the fluorescence reception intensity corresponding to the fluorescence pixel points according to the depth information corresponding to the fluorescence pixel points, the reference depth, and the reference fluorescence reception intensity includes: Determine a second ratio between the square of the reference depth and the square of the depth information corresponding to the fluorescent pixel, and determine a third ratio between the second ratio and a preset adjustment factor; Determine a second product between the third ratio and the reference fluorescence reception intensity, and determine a change amount of the fluorescence reception intensity corresponding to the fluorescent pixel as the second difference between the reference fluorescence reception intensity and the second product; Correspondingly, the correcting the fluorescence reception intensity of the fluorescent pixel according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image includes: Correct the fluorescence reception intensity of the fluorescent pixel to the sum of the fluorescence reception intensity of the fluorescent pixel and the change amount of the fluorescence reception intensity, so as to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

7. The endoscopic imaging method according to claim 1, wherein The method further includes: Fuse the first white light image and the corrected first fluorescence image to obtain a first fused image, and fuse the second white light image and the corrected second fluorescence image to obtain a second fused image; Determine a target image according to the first fused image and the second fused image, and output the target image.

8. The endoscopic imaging method according to claim 7, wherein The determining the target image according to the first fused image and the second fused image includes: Determine the first fused image or the second fused image as the target image.

9. The endoscopic imaging method according to claim 7, wherein The determining the target image according to the first fused image and the second fused image includes: Generate a stereoscopic image as the target image according to the first fused image and the second fused image.

10. The endoscopic imaging method according to claim 1, characterized in that, After acquiring the first white light image and the first fluorescence image of the object through the first image sensor, and acquiring the second white light image and the second fluorescence image of the object through the second image sensor, it further includes: Perform a demosaicing operation on the first white light image and the second white light image.

11. An endoscopic imaging device, characterized in that, Includes: An acquisition unit, configured to acquire a first white light image and a first fluorescence image of an object through a first image sensor, and acquire a second white light image and a second fluorescence image of the object through a second image sensor; A first determination unit, configured to determine the depth information corresponding to the fluorescent pixel in the first fluorescence image and / or the second fluorescence image according to the first white light image and the second white light image; A second determination unit, configured to determine a reference depth and determine a reference fluorescence reception intensity corresponding to the reference depth; A third determination unit, configured to determine a change amount of the fluorescence reception intensity corresponding to the fluorescent pixel according to the depth information corresponding to the fluorescent pixel, the reference depth, and the reference fluorescence reception intensity; A correction unit, configured to correct the fluorescence reception intensity of the fluorescent pixel according to the change amount of the fluorescence reception intensity to obtain a corrected first fluorescence image and / or a corrected second fluorescence image.

12. An electronic device, characterized in that, Includes: A memory, configured to store a computer program; A processor, configured to implement the steps of the endoscope imaging method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the endoscopic imaging method according to any one of claims 1 to 10 are implemented.

Images