Endoscope equipment and image fusion method

Abstract

The invention relates to the technical field of image processing, in particular to endoscope equipment and an image fusion method. According to the endoscope equipment and the image fusion method provided by the invention, the current frame of white light image and the current frame of fluorescence image of the target detection part are collected, and after it is determined that the lens of the endoscope equipment moves, the current frame of fluorescence image is corrected according to the brightness change information of the current frame of white light image and the previous frame of white light image. Since the brightness change of the corresponding area of the white light image is positively correlated with the brightness change of the corresponding area of the fluorescent image, the fluorescent edge information of the fluorescent image is enhanced according to the brightness change information between the white light image frames, and the definition of the fluorescent edge of the target image after fusion processing is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an endoscope device and an image fusion method. Background Art

[0002] At present, endoscopic equipment has become one of the important tools for humans to identify organ lesions. Its clinical application is becoming more and more popular and it is a very widely used medical device.

[0003] The white light images and fluorescence images collected by the endoscopic equipment have their own characteristics in their respective spectral ranges. The target image generated by the fusion of the two can help doctors make accurate and quick judgments.

[0004] However, during the use of the endoscope, the movement of the fluorescence irradiation area caused by the movement of the lens of the endoscope will affect the clarity of the fluorescence edge of the target image. Summary of the invention

[0005] In order to solve the above-mentioned problems in the prior art, the embodiments of the present application provide an endoscope device and an image fusion method for improving the clarity of the fluorescent edge in the target image after fusion processing.

[0006] In a first aspect, an embodiment of the present application provides an endoscope device, the endoscope device comprising a lens, a processor, a white light camera, a fluorescent camera, and a display;

[0007] The processor is configured to:

[0008] Collecting a current frame white light image and a current frame fluorescence image of the target detection part, wherein the current frame white light image is an image containing the target detection part collected by the white light camera at the current moment, and the fluorescence image is an image containing the target detection part collected by the fluorescence camera at the current moment;

[0009] After determining that the lens has moved, the current frame of fluorescence image is corrected according to brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image;

[0010] Performing image fusion on the corrected current frame fluorescence image and the current frame white light image to obtain a fused target image;

[0011] The display is configured to display the target image after the fusion process.

[0012] In a possible implementation manner, when the processor determines that the lens moves, the processor is specifically configured to:

[0013] Determining the exposure amount of the current frame of white light image, and determining the exposure amount of the previous frame of white light image;

[0014] If the exposure variation between the current frame white light image and the previous frame white light image is greater than a first threshold, it is determined that the lens moves.

[0015] In a possible implementation manner, when the processor determines the exposure amount of the current frame of white light image, it is specifically configured as follows:

[0016] Determine a plurality of first exposure regions in the current frame of white light image according to a first region determination rule corresponding to the target detection part; determine an exposure amount of the current frame of white light image according to a first brightness value of each of the first exposure regions;

[0017] When the processor determines the exposure amount of the previous frame of white light image, it is specifically configured as follows:

[0018] According to the first area determination rule corresponding to the target detection part, multiple second exposure areas in the previous frame of white light image are determined; according to the second brightness value of each of the second exposure areas, the exposure amount of the previous frame of white light image is determined.

[0019] In a possible implementation, when the processor determines the brightness change information of the current frame white light image and the previous frame white light image, it is specifically configured as follows:

[0020] Determining a plurality of third exposure regions in the current frame of white light image and a plurality of fourth exposure regions in the previous frame of white light image according to a second region determination rule corresponding to the target detection part;

[0021] The brightness change information is determined according to the third brightness value of each of the third exposure areas and the fourth brightness value of each of the fourth exposure areas.

[0022] In a possible implementation, when the processor determines the brightness change information according to the third brightness value of each of the third exposure areas and the fourth brightness value of the fourth exposure area corresponding to the third exposure area, the processor is specifically configured as follows:

[0023] The following operations are performed for each of the third exposure areas: determining a gradient change value according to a third brightness value of the third exposure area and a fourth brightness value of a target fourth exposure area, wherein the target fourth exposure area is at the same position as the third exposure area in the previous frame of white light image and in the current frame of white light image;

[0024] A gradient change matrix is ​​generated according to each gradient change value, and the gradient change matrix is ​​used as the brightness change information.

[0025] In a possible implementation manner, when the processor corrects the current frame of fluorescence image according to the brightness change information of the current frame of white light image and the previous frame of white light image, the processor is specifically configured as follows:

[0026] Convert the current frame fluorescence image to the HSV color domain to obtain a fluorescence image to be corrected;

[0027] The following operations are performed for each pixel of the fluorescent image to be corrected: the brightness value of the pixel is corrected according to the brightness change information, and the saturation value of the pixel is corrected according to the brightness change information and the corrected brightness value of the pixel.

[0028] In a possible implementation manner, when the processor corrects the brightness value of the pixel point according to the brightness change information, the processor is specifically configured as follows:

[0029] The brightness value of the pixel point is filtered according to each gradient change information in the gradient change matrix to obtain the brightness value of the pixel point corresponding to the pixel point in the corrected fluorescent image.

[0030] In a possible implementation manner, when the processor corrects the saturation value of the pixel point according to the brightness change information and the corrected brightness value of the pixel point, the processor is specifically configured as follows:

[0031] Normalizing the corrected brightness value of the pixel to obtain a correction coefficient for the saturation value of the pixel;

[0032] Filtering the saturation value of the pixel point according to each gradient change information in the gradient change matrix to obtain a filtered saturation value;

[0033] The saturation value of a pixel point corresponding to the pixel point in the corrected fluorescence image is determined according to the filtered saturation value and the correction coefficient.

[0034] In a possible implementation manner, when the processor performs image fusion on the corrected fluorescence image and the current frame white light image, the processor is specifically configured as follows:

[0035] Convert the current frame white light image to the HSV color domain to obtain a white light image to be fused;

[0036] The following operations are performed for each pixel of the corrected fluorescence image: if the brightness value of the pixel is not greater than a second threshold, the pixel information of the pixel corresponding to the pixel in the target image is determined according to the pixel information of the pixel corresponding to the pixel in the white light image to be fused; if the brightness value of the pixel is greater than the second threshold, the color information of the pixel corresponding to the pixel in the target image is determined according to the pixel information of the pixel corresponding to the pixel in the white light image to be fused and the pixel information of the pixel in the corrected fluorescence image.

[0037] In a second aspect, an embodiment of the present application provides an image fusion method, the method comprising:

[0038] Collecting a current frame white light image and a current frame fluorescence image of the target detection part, wherein the current frame white light image is an image containing the target detection part collected by the white light camera at the current moment, and the fluorescence image is an image containing the target detection part collected by the fluorescence camera at the current moment;

[0039] After determining that the lens has moved, the current frame of fluorescence image is corrected according to brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image;

[0040] The corrected current frame fluorescence image and the current frame white light image are fused to obtain a fused target image.

[0041] In a third aspect, an embodiment of the present application provides an image fusion device, including:

[0042] An acquisition module, used to acquire a current frame white light image and a current frame fluorescence image of a target detection part, wherein the current frame white light image is an image including the target detection part acquired by the white light camera at the current moment, and the fluorescence image is an image including the target detection part acquired by the fluorescence camera at the current moment;

[0043] A correction module, for correcting the current frame of fluorescence image according to brightness change information of the current frame of white light image and the previous frame of white light image after determining that the lens has moved, to obtain a corrected current frame of fluorescence image;

[0044] The fusion module is used to fuse the corrected current frame fluorescence image and the current frame white light image to obtain a fused target image.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image fusion method is implemented.

[0046] In the embodiment of the present application, the current frame white light image and the current frame fluorescence image of the target detection part are collected, and after determining that the lens of the endoscope device has moved, the current frame fluorescence image is corrected according to the brightness change information of the current frame white light image and the previous frame white light image. Since the brightness changes of the corresponding areas of the white light image and the fluorescence image are positively correlated, the present application enhances the fluorescence edge information of the fluorescence image according to the brightness change information between the white light image frames, thereby improving the clarity of the fluorescence edge of the target image after the fusion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A schematic diagram of the structure of an endoscope device provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of an image fusion method provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of an image fusion method provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of an exposure area provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of an exposure area provided in an embodiment of the present application;

[0053] Figure 6 A flowchart of an image fusion method provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of an exposure area provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of an exposure area provided in an embodiment of the present application;

[0056] Fig. 9 A flowchart of an image fusion method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, “and” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0059] Specifically, in the description of the present application, it needs to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application, and, unless otherwise specified, the "connection" and "connection" mentioned in the present application include direct and indirect connections (connections).

[0060] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0061] Figure 1 The schematic diagram of the structure of an endoscope device in an embodiment of the present application is shown as an example. Figure 1As shown, the endoscope device includes a light source 1, a light guide 2, a mirror rod 3, a filter 4, a lens 5, a dichroic beam splitter 6, a white light camera 7, a fluorescent camera 8, a white light camera control module 9, a fluorescent camera control module 10, a processor 11 and a display 12. Among them, the front end of the mirror rod 3 is provided with a lens (not shown in the figure), the processor 11 is respectively connected to the white light camera control module 9, the fluorescent camera control module 10, and the display 12, the fluorescent camera control module 10 is connected to the fluorescent camera 8, and the white light camera control module 9 is connected to the white light camera 7.

[0062] The light source 1 can emit excitation light and white light, which are transmitted through the light guide 2 and coupled into the mirror rod 3; the excitation light and white light are emitted from the lens arranged at the front end of the mirror rod 3 and reach the detection part, and the excitation light, white light and fluorescence reflected by the detection part are collected by the mirror rod 3; the excitation light is filtered out by the filter 4, and the fluorescence and white light are focused by the lens 5 through the filter 4; the white light is imaged on the white light camera 7 through the dichroic beam splitter 6 to obtain a white light image; the fluorescence is reflected by the dichroic beam splitter 6 and imaged on the fluorescence camera 8 to obtain a fluorescence image. After obtaining the white light image of the current frame and the white light image of the previous frame, the white light camera control module 9 can determine the exposure change of the white light image and transmit the exposure change to the processor 11. The processor 11 receives the exposure change of the white light image and can determine whether to correct the fluorescence image collected by the fluorescence camera 8 according to the exposure change of the white light image. If the processor 11 needs to correct the fluorescence image, after correcting the fluorescence image, the corrected fluorescence image and the white light image are fused to obtain a target image after fusion processing, and the target image signal is output to the display 12. If the processor 11 does not need to correct the fluorescence image, the fluorescence image and the white light image are directly fused to obtain a target image after fusion processing, and the target image signal is output to the display 12.

[0063] It should be noted that, in another optional embodiment, the white light camera control module and the fluorescence camera control module may not be provided in the endoscope device, and the white light camera and the fluorescence camera may be directly controlled by the processor in the endoscope device, and the image fusion method in the embodiment of the present application may be executed by the processor in the endoscope device.

[0064] At present, endoscope equipment has become one of the important tools for humans to identify organ lesions. Its clinical application is becoming more and more popular and it is a widely used medical device. The white light image and fluorescent image collected by the endoscope have their own characteristics in their own spectral ranges. The target image generated by the fusion of the two can help doctors make accurate and quick judgments. However, during the use of the endoscope, the movement of the fluorescent irradiation area caused by the movement of the lens of the endoscope will affect the clarity of the fluorescent edge of the target image.

[0065] Based on this, the embodiment of the present application provides an endoscope device and an image fusion method, which collects the current frame white light image and the current frame fluorescence image of the target detection part, and after determining that the lens of the endoscope device has moved, the current frame fluorescence image is corrected according to the brightness change information of the current frame white light image and the previous frame white light image. Since the brightness changes of the corresponding areas of the white light image and the fluorescence image are positively correlated, the present application enhances the fluorescence edge information of the fluorescence image according to the brightness change information between the white light image frames, thereby improving the clarity of the fluorescence edge of the target image after the fusion process, so that doctors can perform more accurate surgical planning.

[0066] Figure 2 A flowchart of an image fusion method provided in an embodiment of the present application is shown. The method can be applied to Figure 1 In the processor shown, Figure 2 As shown, the method may include the following steps:

[0067] Step S201 , collecting a current frame white light image and a current frame fluorescence image of the target detection part.

[0068] The current frame white light image is an image containing the target detection part collected by the white light camera at the current moment, and the fluorescence image is an image containing the target detection part collected by the fluorescence camera at the current moment.

[0069] In the embodiment of the present application, the white light emitted by the light source in the endoscope device irradiates the target detection part, and the white light reflected by the target detection part is imaged on the white light camera, and the white light image of the current frame can be obtained. After the excitation light emitted by the light source in the endoscope device irradiates the target detection part with the contrast agent, the fluorescence image of the current frame can be collected by the fluorescence camera. The white light image and the fluorescence image can be acquired synchronously.

[0070] Step S202, after determining that the lens has moved, the current frame of fluorescence image is corrected according to the brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image.

[0071] Since the movement of the lens of the endoscope device will cause the movement of the fluorescent irradiation area, resulting in unclear fluorescent edges of the current frame fluorescent image and the target image, in the embodiment of the present application, it is possible to first determine whether the lens of the endoscope device has moved, and then determine whether to correct the current frame fluorescent image.

[0072] In an optional embodiment, the method for determining the lens movement of the endoscope device may include: Figure 3 The following steps are shown:

[0073] Step S2021, determining the exposure amount of the current frame of white light image, and determining the exposure amount of the previous frame of white light image.

[0074] In the embodiment of the present application, when determining the exposure amount of the current frame white light image, a rule can be first determined according to the first area corresponding to the target detection part to determine multiple first exposure areas in the current frame white light image.

[0075] Specifically, the first area determination rule may be a correspondence between the detection part and the first exposure area, and different detection parts may correspond to first exposure areas at different positions. The first exposure area corresponds to the second exposure area described below. In implementation, the first exposure area is determined according to the actual use scenario, and its exposure amount is higher than the first set threshold. For example, assuming that the current target detection part is the liver, the liver may correspond to Figure 4 The five first exposure areas are shown.

[0076] It should be noted that the present application is only described by taking five first exposure areas as an example. In other embodiments, fewer or more first exposure areas may be selected to determine the exposure amount of the current frame white light image. For example, Figure 5 The nine first exposure areas shown are used to determine the exposure amount of the current frame of white light image, and this application does not impose any limitation on this.

[0077] After a plurality of first exposure regions are determined in the current frame of white light image, the exposure amount of the current frame of white light image may be determined according to the first brightness value of each first exposure region.

[0078] In an optional implementation, the brightness mean of each pixel point in each first exposure area in the current frame white light image can be determined to obtain the first brightness value of each first exposure area. Then, the weighted average of the first brightness values ​​of each first exposure area is used as the exposure value of the current frame white light image.

[0079] Specifically, the following formula (1) can be used to determine the average brightness of the pixels in each first exposure area in the current frame white light image. Formula (1) can be expressed as:

[0080]

[0081] Among them, L region represents the average brightness of each pixel in the first exposure area, L pixel represents the brightness value of each pixel in the first exposure area, and N represents the number of pixels in the first exposure area.

[0082] Furthermore, the following formula (2) can be used to determine the weighted average of the first brightness values ​​of each first exposure area in the current frame white light image, and the weighted average is used as the exposure value of the current frame white light image. Formula (2) can be expressed as:

[0083]

[0084] Among them, L sum represents the weighted average of the first brightness values ​​of each first exposure area, that is, the exposure amount of the current frame white light image, L region represents the average brightness of each pixel in the first exposure area, ω region represents the contribution weight of each first exposure area, and M represents the number of first exposure areas.

[0085] In an embodiment of the present application, when determining the exposure amount of the previous frame of white light image, the multiple second exposure areas in the previous frame of white light image can be determined according to the first area determination rule corresponding to the target detection part. Then, the exposure amount of the previous frame of white light image is determined according to the second brightness value of each second exposure area. Among them, the specific steps of determining the multiple second exposure areas in the previous frame of white light image according to the first area determination rule corresponding to the target detection part, and determining the exposure amount of the previous frame of white light image according to the second brightness value of each second exposure area are the same as the process of determining the multiple first exposure areas in the current frame of white light image according to the first area determination rule corresponding to the target detection part, and determining the exposure amount of the current frame of white light image according to the first brightness value of each first exposure area, and will not be repeated here.

[0086] Step S2022: if the exposure change between the current frame white light image and the previous frame white light image is greater than a first threshold, it is determined that the lens moves.

[0087] After determining the exposure of the current frame of white light image and the exposure of the previous frame of white light image, the exposure change between the current frame of white light image and the previous frame of white light image can be determined, and it is determined whether the exposure change is greater than a preset first threshold. If it is determined that the exposure change between the current frame of white light image and the previous frame of white light image is not greater than the first threshold, it is determined that the lens has not moved. If it is determined that the exposure change between the current frame of white light image and the previous frame of white light image is greater than the preset first threshold, it is determined that the lens has moved.

[0088] Specifically, the exposure change between the current frame white light image and the previous frame white light image can be determined according to the following formula (3), which can be expressed as:

[0089] |L currentsum -L presum |=L change ————Formula (3)

[0090] Among them, L currentsum Indicates the exposure of the current frame of white light image, L presum Indicates the exposure of the previous frame of white light image, L change Indicates the exposure change between the current frame white light image and the previous frame white light image.

[0091] After determining that the lens has moved, the current frame of fluorescence image can be corrected according to the brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image.

[0092] In the embodiment of the present application, the method for determining the brightness change information of the current frame white light image and the previous frame white light image may include: Figure 6 The following steps are shown:

[0093] Step S601 , determining a plurality of third exposure regions in a current frame of white light image and a plurality of fourth exposure regions in a previous frame of white light image according to a second region determination rule corresponding to a target detection part.

[0094] In an embodiment of the present application, when determining the brightness change information of the current frame white light image and the previous frame white light image, a rule can be first determined based on the second area corresponding to the target detection part to determine multiple third exposure areas in the current frame white light image and determine multiple fourth exposure areas in the previous frame white light image.

[0095] Specifically, the second area determination rule may be a correspondence between the detection part and the third exposure area, and different detection parts may correspond to third exposure areas at different positions. The third exposure area and the fourth exposure area correspond one to one. In implementation, the third exposure area may be determined according to the actual use scenario, and its exposure amount is higher than the second set threshold. For example, assuming that the current target detection part is the liver, the liver may correspond to Figure 7 The 4 third exposure areas are shown.

[0096] It should be noted that the present application is described by taking four third exposure areas as an example. In other implementations, fewer or more third exposure areas may be selected to determine the exposure amount of the current frame white light image. For example, Figure 8 The eight third exposure areas shown are not limited in this application.

[0097] Step S602, performing the following operations for each third exposure area: determining a gradient change value according to a third brightness value of the third exposure area and a fourth brightness value of the target fourth exposure area.

[0098] The position of the target fourth exposure area in the previous frame of white light image is the same as the position of the third exposure area in the current frame of white light image.

[0099] In the specific implementation, let’s assume Figure 7 The four exposure areas shown in the figure are exposure area up, exposure area down, exposure area left and exposure area right.

[0100] For the exposure area up in the previous frame of white light image, the average brightness of the pixels in the exposure area up in the previous frame of white light image can be determined, and then, for the exposure area up in the current frame of white light image, the average brightness of the pixels in the exposure area up in the current frame of white light image is determined. Next, the difference between the average brightness of the pixels in the exposure area up in the previous frame of white light image and the average brightness of the pixels in the exposure area up in the current frame of white light image is determined, and finally, the ratio of the difference to the average brightness of the pixels in the exposure area up in the previous frame of white light image is used as the gradient change value corresponding to the exposure area up.

[0101] Specifically, the gradient change value corresponding to the exposure area up can be determined according to the following formula (4), which can be expressed as:

[0102]

[0103] in, Indicates the average brightness of the pixels in the exposure area up in the previous frame of white light image. Indicates the average brightness of the pixels in the exposure area up in the current frame of white light image.

[0104] For the exposure area down in the previous frame of white light image, the average brightness of the pixels in the exposure area down in the previous frame of white light image can be determined, and then, for the exposure area down in the current frame of white light image, the average brightness of the pixels in the exposure area down in the current frame of white light image is determined, and then, the difference between the average brightness of the pixels in the exposure area down in the previous frame of white light image and the average brightness of the pixels in the exposure area down in the current frame of white light image is determined, and finally, the ratio of the difference to the average brightness of the pixels in the exposure area down in the previous frame of white light image is used as the gradient change value corresponding to the exposure area down.

[0105] Specifically, the gradient change value corresponding to the exposure area down can be determined according to the following formula (5), which can be expressed as:

[0106]

[0107] in, Indicates the average brightness of the pixels in the down exposure area of ​​the previous frame of white light image. Indicates the average brightness of the pixels in the down exposure area of ​​the current frame white light image.

[0108] For the exposure area left in the previous frame of white light image, the average brightness of the pixels in the exposure area left in the previous frame of white light image can be determined, and then, for the exposure area left in the current frame of white light image, the average brightness of the pixels in the exposure area left in the current frame of white light image can be determined. Then, the difference between the average brightness of the pixels in the exposure area left in the previous frame of white light image and the average brightness of the pixels in the exposure area left in the current frame of white light image is determined. Finally, the ratio of the difference to the average brightness of the pixels in the exposure area left in the previous frame of white light image is used as the gradient change value corresponding to the exposure area left.

[0109] Specifically, the gradient change value corresponding to the exposure area left can be determined according to the following formula (6), which can be expressed as:

[0110]

[0111] in, Indicates the average brightness of the pixels in the exposed area left in the previous frame of white light image. Indicates the average brightness of the pixels in the exposed area left in the current frame of white light image.

[0112] For the exposure area right in the previous frame of white light image, the average brightness of the pixels in the exposure area right in the previous frame of white light image can be determined, and then, for the exposure area right in the current frame of white light image, the average brightness of the pixels in the exposure area right in the current frame of white light image can be determined. Then, the difference between the average brightness of the pixels in the exposure area right in the previous frame of white light image and the average brightness of the pixels in the exposure area right in the current frame of white light image is determined, and finally, the ratio of the difference to the average brightness of the pixels in the exposure area right in the previous frame of white light image is used as the gradient change value corresponding to the exposure area right.

[0113] Specifically, the gradient change value corresponding to the exposure area right can be determined according to the following formula (7), which can be expressed as:

[0114]

[0115] in, Indicates the average brightness of the pixels in the exposure area right in the previous frame of white light image. Indicates the average brightness of the pixels in the exposure area right in the current frame of white light image.

[0116] Step S603: Generate a gradient change matrix according to each gradient change value, and use the gradient change matrix as brightness change information.

[0117] After determining each gradient change value corresponding to each exposure area in the white light image, a gradient change matrix can be generated according to each gradient change value.

[0118] For example, continue with Figure 7 Taking the four exposure areas shown as an example, according to the gradient change value corresponding to each exposure area, the gradient change matrix can be determined as follows:

[0119]

[0120] In a specific implementation, the gradient change matrix can be used as the brightness change information of the white light image of the current frame and the white light image of the previous frame. However, it should be noted that the above is explained with 4 exposure areas as an example. If 8 exposure areas are selected, the gradient change matrix can also be expressed as:

[0121]

[0122] In other embodiments, more exposure areas may be selected to generate a higher-order gradient change matrix. For example, a 5×5 gradient change matrix may be generated based on more exposure areas, and this application does not impose any limitation on this.

[0123] After determining the brightness change information of the current frame white light image and the previous frame white light image, the current frame fluorescence image can be corrected according to the brightness change information of the current frame white light image and the previous frame white light image to obtain a corrected current frame fluorescence image.

[0124] In the embodiment of the present application, when correcting the current frame of fluorescent image according to the brightness change information of the current frame of white light image and the previous frame of white light image, the following steps may be included: Fig. 9 The following steps are shown:

[0125] Step S901, converting the current frame fluorescence image into the HSV color domain to obtain the fluorescence image to be corrected.

[0126] Specifically, for each pixel point of the current frame fluorescence image, the following operations may be performed respectively, and the color value corresponding to the target fluorescence color is used as the color value of the pixel point corresponding to the pixel point in the fluorescence image to be corrected.

[0127] For example, assuming that the fluorescence color is to be green, the color value of the pixel corresponding to the pixel in the fluorescence image to be corrected can be set to 60.

[0128] Furthermore, the brightness value of the pixel in the current frame fluorescence image can be used as the saturation value of the pixel corresponding to the pixel in the fluorescence image to be corrected, and the brightness value of the pixel in the current frame fluorescence image can be used as the brightness value of the pixel corresponding to the pixel in the fluorescence image to be corrected.

[0129] In order to retain the fluorescence concentration information, the brightness value and saturation value of each pixel of the fluorescence image to be corrected may be set to the brightness value of the current frame of the fluorescence image.

[0130] Step S902 , performing the following operations for each pixel of the fluorescent image to be corrected: correcting the brightness value of the pixel according to the brightness change information.

[0131] Specifically, the brightness value of the pixel point may be filtered according to each gradient change information in the gradient change matrix to obtain the brightness value of the pixel point corresponding to the pixel point in the corrected fluorescent image.

[0132] Step S903 , performing the following operations for each pixel of the fluorescent image to be corrected: correcting the saturation value of the pixel according to the brightness change information and the corrected brightness value of the pixel.

[0133] In an optional implementation, the corrected brightness value of the pixel point may be normalized first to obtain a correction coefficient of the saturation value of the pixel point.

[0134] Next, the saturation value of the pixel point may be filtered according to each gradient change information in the gradient change matrix to obtain a filtered saturation value.

[0135] Finally, the saturation value of the pixel point corresponding to the pixel point in the corrected fluorescence image can be determined according to the filtered saturation value and the correction coefficient.

[0136] Specifically, the product of the filtered saturation value and the correction coefficient may be used as the saturation value of the pixel point corresponding to the pixel point in the corrected fluorescence image.

[0137] The order of step S902 and step S903 can be exchanged to obtain a corrected fluorescent image with different display effects. If step S902 is performed first and then step S903, a softer display effect can be obtained. Conversely, if step S903 is performed first and then step S902, a sharper display effect can be obtained. In specific implementation, it can be set according to different usage requirements.

[0138] Step S203 , fusing the corrected current frame fluorescence image and the current frame white light image to obtain a fused target image.

[0139] In a specific implementation, the current frame white light image may be first converted to the HSV color domain to obtain the white light image to be fused.

[0140] Then, the following operations are performed for each pixel of the corrected fluorescence image: if it is determined that the brightness value of the pixel is not greater than the second threshold, the pixel information of the pixel corresponding to the pixel in the target image can be determined based on the pixel information of the pixel corresponding to the pixel in the white light image to be fused.

[0141] Exemplarily, assuming that the brightness value of pixel point F in the corrected fluorescence image is not greater than the second threshold, the color value, saturation value and brightness value of the pixel point corresponding to pixel point F in the white light image to be fused can be used as the color value, saturation value and brightness value of the pixel point corresponding to the pixel point in the target image.

[0142] If the brightness value of the pixel is greater than the second threshold, the color information of the pixel corresponding to the pixel in the target image is determined based on the pixel information of the pixel corresponding to the pixel in the white light image to be fused and the pixel information of the pixel in the corrected fluorescence image.

[0143] Exemplarily, assuming that the brightness value of pixel point P in the corrected fluorescence image is greater than the second threshold, the color value of pixel point P can be set to the color value corresponding to the target fluorescence color, for example, it can be set to 60, and the weighted sum of the saturation values ​​of pixel point P in the corrected fluorescence image and the white light image to be fused is used as the saturation value of the pixel point corresponding to pixel point P in the target image, and the weighted sum of the brightness values ​​of pixel point P in the corrected fluorescence image and the white light image to be fused is used as the brightness value of the pixel point corresponding to pixel point P in the target image.

[0144] Specifically, the saturation value of the corresponding pixel point in the target image can be determined according to the following formula (8), which can be expressed as:

[0145]

[0146] in, is the saturation value of the corresponding pixel in the corrected fluorescence image.

[0147] Furthermore, the brightness value of the corresponding pixel in the target image can be determined according to the following formula (9), which can be expressed as:

[0148]

[0149] in, The brightness value of the corresponding pixel in the corrected fluorescence image.

[0150] At this point, the image fusion of the corrected fluorescence image and the white light image is completed.

[0151] Step S204: display the target image after the fusion process.

[0152] After the corrected fluorescence image and the white light image are fused to obtain the target image, the target image can be converted into RGB color gamut output.

[0153] In the embodiment of the present application, the movement of the light source is estimated by the exposure change between white light image frames, and the brightness of the fluorescent image is corrected according to the gradient change matrix to reduce the impact of the light source change on the edge and range of the fluorescent image. At the same time, the fluorescence regions are enhanced to reduce the interference of stray light, improve the saturation performance according to the fluorescence brightness, and fuse the saturation and brightness separately in a normalized manner, thereby improving the accuracy of the fluorescence and the texture transparency of the fused image.

[0154] Based on the same inventive concept, the image fusion method described above in the present application can also be implemented by an image fusion device. The effect of the image fusion device is similar to that of the above method, and will not be described in detail here.

[0155] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in the embodiment of the present application. The principle of solving the problem by the electronic device is similar to the method of the above-mentioned embodiment. Therefore, the implementation of the electronic device can refer to the implementation of the above-mentioned method, and the repeated parts will not be repeated.

[0156] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the image fusion method described in any embodiment of the present application.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An endoscope device, characterized in that: The endoscope device includes a lens, a processor, a white light camera, a fluorescent camera and a display; The processor is configured to: Collecting a current frame white light image and a current frame fluorescence image of the target detection part, wherein the current frame white light image is an image containing the target detection part collected by the white light camera at the current moment, and the fluorescence image is an image containing the target detection part collected by the fluorescence camera at the current moment; After determining that the lens has moved, the current frame of fluorescence image is corrected according to brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image; Performing image fusion on the corrected current frame fluorescence image and the current frame white light image to obtain a fused target image; The display is configured to display the target image after the fusion process.

2. The endoscope device according to claim 1, characterized in that: When the processor determines that the lens moves, the processor is specifically configured to: Determining the exposure amount of the current frame of white light image, and determining the exposure amount of the previous frame of white light image; If the exposure variation between the current frame white light image and the previous frame white light image is greater than a first threshold, it is determined that the lens moves.

3. The endoscope device according to claim 2, characterized in that: When the processor determines the exposure amount of the current frame white light image, it is specifically configured as follows: Determine a plurality of first exposure regions in the current frame of white light image according to a first region determination rule corresponding to the target detection part; determine an exposure amount of the current frame of white light image according to a first brightness value of each of the first exposure regions; When the processor determines the exposure amount of the previous frame of white light image, it is specifically configured as follows: According to the first area determination rule corresponding to the target detection part, multiple second exposure areas in the previous frame of white light image are determined; according to the second brightness value of each of the second exposure areas, the exposure amount of the previous frame of white light image is determined.

4. The endoscope device according to claim 1, characterized in that: When the processor determines the brightness change information of the current frame white light image and the previous frame white light image, it is specifically configured as follows: Determining a plurality of third exposure regions in the current frame of white light image and a plurality of fourth exposure regions in the previous frame of white light image according to a second region determination rule corresponding to the target detection part; The brightness change information is determined according to the third brightness value of each of the third exposure areas and the fourth brightness value of each of the fourth exposure areas.

5. The endoscope device according to claim 4, characterized in that: When the processor determines the brightness change information according to the third brightness value of each of the third exposure areas and the fourth brightness value of the fourth exposure area corresponding to the third exposure area, the processor is specifically configured as follows: The following operations are performed for each of the third exposure areas: determining a gradient change value according to a third brightness value of the third exposure area and a fourth brightness value of a target fourth exposure area, wherein the target fourth exposure area is at the same position as the third exposure area in the previous frame of white light image and in the current frame of white light image; A gradient change matrix is ​​generated according to each gradient change value, and the gradient change matrix is ​​used as the brightness change information.

6. The endoscope device according to claim 5, characterized in that: When the processor corrects the current frame of fluorescent image according to the brightness change information of the current frame of white light image and the previous frame of white light image, the processor is specifically configured as follows: Convert the current frame fluorescence image to the HSV color domain to obtain a fluorescence image to be corrected; The following operations are performed for each pixel of the fluorescent image to be corrected: the brightness value of the pixel is corrected according to the brightness change information, and the saturation value of the pixel is corrected according to the brightness change information and the corrected brightness value of the pixel.

7. The endoscope device according to claim 6, characterized in that: When the processor corrects the brightness value of the pixel point according to the brightness change information, the processor is specifically configured as follows: The brightness value of the pixel point is filtered according to each gradient change information in the gradient change matrix to obtain the brightness value of the pixel point corresponding to the pixel point in the corrected fluorescent image.

8. The endoscope device according to claim 6, characterized in that: When the processor corrects the saturation value of the pixel point according to the brightness change information and the corrected brightness value of the pixel point, the processor is specifically configured as follows: Normalizing the corrected brightness value of the pixel to obtain a correction coefficient for the saturation value of the pixel; Filtering the saturation value of the pixel point according to each gradient change information in the gradient change matrix to obtain a filtered saturation value; The saturation value of a pixel point corresponding to the pixel point in the corrected fluorescence image is determined according to the filtered saturation value and the correction coefficient.

9. The endoscope device according to any one of claims 1 to 8, characterized in that: When the processor performs image fusion on the corrected fluorescence image and the current frame white light image, the processor is specifically configured as follows: Convert the current frame white light image to the HSV color domain to obtain a white light image to be fused; For each pixel point of the corrected fluorescence image, the following operations are performed respectively: if the brightness value of the pixel point is not greater than a second threshold, determining the pixel information of the pixel point corresponding to the pixel point in the target image according to the pixel information of the pixel point corresponding to the pixel point in the white light image to be fused; If the brightness value of the pixel point is greater than the second threshold, the color information of the pixel point corresponding to the pixel point in the target image is determined based on the pixel information of the pixel point corresponding to the pixel point in the white light image to be fused and the pixel information of the pixel point in the corrected fluorescence image.

10. An image fusion method, characterized in that: The method comprises: Collecting a current frame white light image and a current frame fluorescence image of the target detection part, wherein the current frame white light image is an image containing the target detection part collected by the white light camera at the current moment, and the fluorescence image is an image containing the target detection part collected by the fluorescence camera at the current moment; After determining that the lens has moved, the current frame of fluorescence image is corrected according to brightness change information of the current frame of white light image and the previous frame of white light image to obtain a corrected current frame of fluorescence image; The corrected current frame fluorescence image and the current frame white light image are fused to obtain a fused target image.

Images