A method and system for fusing fluorescence image and video image

Abstract

The present invention belongs to the field of image processing technology, and provides a method and system for fusing fluorescence images with video images, including: obtaining several frames of continuous video images and several continuous fluorescence images within a time period; for all video images between two adjacent fluorescence images, respectively calculating the focus value of the reference column of each frame of video image, and selecting the video image with the largest focus value as the optimal image; after converting all pixels in the optimal image and the fluorescence image to the same coordinate system, fusing the optimal image and the fluorescence image to obtain a fused image; the step of calculating the focus value includes: for the reference column of the video image, calculating the difference in grayscale values ​​between two adjacent pixels; and taking the sum of all differences as the focus value. Fluorescence images can be displayed in real time in the video, breaking through the limitation of single-image fluorescence imaging, and effectively avoiding the influence of unclear video images on the fusion effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for fusing a fluorescence image and a video image. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Endoscope is a detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, software, etc. It has image sensors, optical lenses, light sources, mechanical devices, etc. The endoscope enters the stomach through the mouth or other natural channels into the body, and can directly see lesions that cannot be displayed by external X-rays. In modern medicine, it is an important method for disease diagnosis and treatment. Due to its advantages such as non-invasiveness, sensitivity and rapidity, endoscopes are increasingly widely used in clinical applications, especially in the diagnosis of gastrointestinal cancer.

[0004] However, the current detection rate and diagnostic ability of white light endoscopy for early lesions are limited and mainly rely on the experience of clinicians. Fluorescence endoscopy can assist doctors in diagnosis, but the current auxiliary diagnosis of fluorescence endoscopy only stays on taking a single fluorescent image, which limits the diagnosis area, increases the diagnosis time, reduces the inspection efficiency, makes it difficult to provide diagnosis and treatment strategies in real time, and worsens the patient's medical experience.

[0005] At present, there are methods for fusing fluorescence images with visible light images, which can display fluorescence images in visible light images, but there are certain limitations. For example, the patent with application number 202210027467.X and invention name "A method and system for fusing visible light images and fluorescence images" requires the registration of visible light cameras and near-infrared cameras to achieve synchronous acquisition of visible light images and fluorescence images in order to achieve the fusion of fluorescence images and visible light images. However, this method requires the synchronous acquisition of visible light images and fluorescence images. However, most of the current white light endoscopes collect videos, while fluorescence endoscopes collect images. That is, video images are collected using a camera with 60 frames, so the number of video images collected per second is 60, while fluorescence images are collected using a high-speed camera, and the number of images that can be collected per second is 30. This situation results in one fluorescence image corresponding to multiple video images, while the prior art only considers the situation where one fluorescence image corresponds to one video image. Moreover, when collecting video images, the white light endoscope moves in real time, and the clarity of each image cannot be guaranteed, which will affect the fusion effect of video images and fluorescence images. Summary of the invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method and system for fusing fluorescence images and video images, which quickly selects the optimal frame video image between two adjacent fluorescence images by adjusting the focus value and fuses it with the fluorescence image, thereby completing the precise fusion of the video image and the fluorescence image and effectively avoiding the influence of the unclear video image on the fusion effect.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a method for fusing a fluorescence image with a video image.

[0009] A method for fusing a fluorescence image and a video image, comprising:

[0010] Acquire a plurality of frames of continuous video images and a plurality of continuous fluorescence images within a period of time;

[0011] For all video images between two adjacent fluorescent images, the focus value of the reference column of each frame of the video image is calculated respectively, and the video image with the largest focus value is selected as the optimal image; the step of calculating the focus value includes: for the reference column of the video image, calculating the difference in grayscale values ​​between two adjacent pixel points; and taking the sum of all the differences as the focus value;

[0012] After all pixels in the optimal image and the fluorescence image are converted to the same coordinate system, the optimal image and the fluorescence image are fused to obtain a fused image; the fusion method is: calculate the average value of the pixel values ​​of the pixel points in the optimal image and the fluorescence image, and use the average value as the pixel value of the pixel points in the fused image.

[0013] Furthermore, for the optimal image and the fluorescence image converted to the same coordinate system, the pixel values ​​of the positions with only a single pixel point in the coordinate system are cleared to zero, and then the optimal image and the fluorescence image are fused.

[0014] Furthermore, the method further includes: arranging all fused images in chronological order.

[0015] Furthermore, the step of converting all pixels in the optimal image and the fluorescence image to the same coordinate system includes: based on multiple calibration points, converting the optimal image and the fluorescence image into two-dimensional coordinate systems respectively; and converting the pixels in the optimal image and the fluorescence image into the same two-dimensional coordinate system through relative position deviation.

[0016] Furthermore, the reference column is a fixed column specified manually.

[0017] Furthermore, the benchmark column is a column with the smallest sum of variances, and the sum of variances is: ,in, represents the sum of variances of the i-th column, , represents the number of pixels in the i-th column of each j-frame video image between two fluorescent images, represents the mean gray value of all pixels in the i-th column of each j-frame video image between two fluorescent images, x ijk Represents the gray value of the kth pixel in the ith column of each j-frame video image between two fluorescent images.

[0018] A second aspect of the present invention provides a system for fusing fluorescence images and video images.

[0019] A fluorescence image and video image fusion system, comprising:

[0020] A data acquisition module is configured to: acquire a plurality of frames of continuous video images and a plurality of continuous fluorescence images within a period of time;

[0021] The image selection module is configured to: for all video images between two adjacent fluorescent images, respectively calculate the focus value of the reference column of each frame of the video image, and select the video image with the largest focus value as the optimal image; the focus value calculation step includes: for the reference column of the video image, calculating the difference in grayscale values ​​between two adjacent pixel points; and taking the sum of all the differences as the focus value;

[0022] The image fusion module is configured to: after converting all pixels in the optimal image and the fluorescent image to the same coordinate system, fuse the optimal image and the fluorescent image to obtain a fused image; the fusion method is: calculate the average value of the pixel values ​​of the pixel points in the optimal image and the fluorescent image, and use the average value as the pixel value of the pixel points in the fused image.

[0023] Furthermore, for the optimal image and the fluorescence image converted to the same coordinate system, the pixel values ​​of the positions with only a single pixel point in the coordinate system are cleared to zero, and then the optimal image and the fluorescence image are fused.

[0024] Furthermore, the method further includes: arranging all fused images in chronological order.

[0025] Furthermore, the step of converting all pixels in the optimal image and the fluorescence image to the same coordinate system includes: based on multiple calibration points, converting the optimal image and the fluorescence image into two-dimensional coordinate systems respectively; and converting the pixels in the optimal image and the fluorescence image into the same two-dimensional coordinate system through relative position deviation.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention quickly selects the optimal frame video image between two adjacent fluorescence images by adjusting the focus value, and merges it with the fluorescence image, thereby completing the precise fusion of the video image and the fluorescence image, so that the patient's fluorescence can be collected in real time during the use of the endoscope, and the influence of unclear video images on the fusion effect is effectively avoided.

[0028] The present invention integrates video images with fluorescent images and can display fluorescent images in real time in the video, breaking through the limitation of single-frame fluorescent imaging, allowing doctors to continuously observe the fluorescent imaging conditions in the patient's body and assisting the doctor's diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0030] Figure 1 is a flow chart of a method for fusing a fluorescence image and a video image shown in the first embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the placement of a video camera, a fluorescent camera and a marker point shown in the first embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of a coordinate system setting shown in Embodiment 1 of the present invention;

[0033] Figure 4 is a schematic diagram of standard calibration points for in vitro calibration shown in the first embodiment of the present invention;

[0034] Figure 5 It is a schematic diagram of the fusion of the fluorescence image and the video image shown in the first embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] In the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention is further described below with reference to the accompanying drawings and embodiments.

[0038] Embodiment 1

[0039] This embodiment provides a method for fusing a fluorescence image and a video image.

[0040] The present embodiment provides a method for fusing fluorescence images and video images, which can fuse a single fluorescence image and a video image to complete the real-time display of the fluorescence area in the video image, thereby improving the detection rate of early lesions in patients, improving the detection efficiency, and assisting doctors in making rapid real-time diagnoses.

[0041] The present embodiment provides a method for fusing fluorescence images and video images, which uses the white light lens and the fluorescence lens integrated on the endoscope to collect fluorescence images and video images, and fuses the fluorescence images and video images through the image fusion method, so that doctors can observe continuous fluorescence information, help doctors diagnose early lesions, improve the detection rate of early lesions, shorten the detection time, improve the detection efficiency, reduce the pain of patients, and improve the patient's medical experience. A real-time lesion identification auxiliary means under endoscopy can be provided to help doctors diagnose and detect, and avoid missed detection and missed diagnosis of digestive tract diseases.

[0042] This embodiment provides a method for fusing a fluorescence image and a video image, such as Figure 1 As shown, the following steps are included:

[0043] Step 1: Acquire a video image containing at least three calibration points and a fluorescence image containing at least three calibration points; and, Figure 2 and Figure 4 As shown, in order to accurately fuse the video image with the fluorescent image, it is necessary to set positioning points (also known as markers or calibration objects or standard calibration points or calibration points or identification points), and use the standard calibration points to calibrate the video camera or the fluorescent camera, that is, to calibrate the video image and the fluorescent image.

[0044] As an implementation method, the structure of the endoscope needs to be modified, and markers are placed on the top of the endoscope. The three markers are not colinear and not coplanar. The three markers are required to appear in the lens acquisition range of the video image and the fluorescent image at the same time, and each frame of the video image contains these three points, and each fluorescent image also contains these three marking points.

[0045] like Figure 4 As shown, calibration point 1, calibration point 2 and calibration point 3 are arranged counterclockwise in sequence, a light guide is included between calibration point 1 and calibration point 2, a clamp is provided between calibration point 2 and calibration point 3, a light guide and a nozzle are provided between calibration point 3 and calibration point 1, and calibration point 1, calibration point 2 and calibration point 3 surround a video lens and a fluorescent lens.

[0046] There are many types of calibration objects, which are not limited and are required to be clearly displayed in both video images and fluorescent images.

[0047] For video images and fluorescent images containing three landmarks, the two images need to be calibrated according to the points in the images (i.e., image calibration). Before using an endoscope to acquire patient images, dual-lens calibration needs to be performed in vitro, using standard two-dimensional coordinate systems and standard points in the coordinate system (a three-dimensional spatial coordinate system, where each point has a specific coordinate (x, y, z)).

[0048] Among them, calibration refers to the calibration of the image. Because the video camera and the fluorescent camera are not at the same point, the lenses of the two cameras can be selected as the origins of two different coordinate systems to generate two three-dimensional coordinate systems. According to the positions of the three calibration objects in the two different three-dimensional coordinate systems, the three-dimensional coordinate systems of the two different cameras can be converted. For example, in the A coordinate system, a=(0, 0, 0), and in the B coordinate system, a=(0, 0, -1), which means that the XY plane is translated upward by one unit.

[0049] like Figure 3 As shown, a video camera and a fluorescent camera are used to collect image information simultaneously. At this time, the fixed camera is required to remain motionless. A frame of the video image and the fluorescent image are selected for dual-lens calibration: a unique two-dimensional coordinate system is generated based on the calibration points in the two images (such as Figure 3 As shown, the coordinate systems calibrated in the two lenses are the same coordinate system (i.e., the only two-dimensional coordinate system), which remains unchanged, and the relative position of the two lenses changes), and the coordinates of the camera lens are read in the two images. According to the coordinate information of the camera lens, the relative position difference of the coordinates in the two images is obtained, and the relative position difference is recorded (a, b). For example, in the only two-dimensional coordinate system, the coordinate system of the center point of the video lens is (x1, y1), and the coordinate system of the center point of the fluorescent camera lens is (x1, y2), then the relative position difference of the coordinates in the two images is (0, y1-y2).

[0050] Step 2: Use the endoscope to simultaneously collect video images and fluorescence images of the patient's body, and save the two types of image information.

[0051] In the present invention, video image acquisition uses a camera with a frame rate of 60 frames, so the number of video images acquired per second is 60, while fluorescence image acquisition uses a high-speed camera, which can acquire 30 images per second. The video image needs to be processed to match the frame rate of the fluorescence image.

[0052] Step 3: Record each frame of video image between every two fluorescent images, and then process multiple frames of video images to select the best frame of video image (such as Figure 5 The leftmost image shown) and the fluorescence image (as Figure 5 The fusion is performed as shown in the middle figure).

[0053] When processing multiple frames of video images, a reference column is selected, and for each frame of video images, a focus value S of the pixels in the reference column is calculated. The step of calculating the focus value includes: for the reference column, calculating the difference in grayscale values ​​between two adjacent pixels; and taking the sum of all differences as the focus value, that is, S=Σ|V i -V i-1 |, where V i represents the gray value of the i-th pixel in the reference column, V i-1 is the grayscale value of the i-1th pixel in the reference column.

[0054] As an implementation method, the selected reference column may be a fixed column designated by a user. This method has a defect, that is, it is difficult to select a column with a relatively balanced gray value, and the focus value calculated based on this column has a deviation.

[0055] As another implementation, the reference column is selected as follows: for the pixels in the i-th column, the gray value x of each frame of the video image between the two fluorescent images is recorded. ijk , where x ijk Represents the gray value of the kth pixel in the i-th column of each j-frame video image between two fluorescent images, and calculates the sum of the variances of each column: , represents the gray value variance of the i-th column of each j-frame video image between two fluorescent images, ,in, represents the number of pixels in the i-th column, Represents the mean of the grayscale values ​​of all pixels in the i-th column; the column with the smallest sum of variance is selected as the reference column. This method uses the sum of the variances of all video images between two fluorescent images to select the reference column, which can effectively avoid the focus value deviation problem caused by the uneven distribution of the grayscale value of the image itself.

[0056] The focus value S is calculated for each of the multiple video frames between the two fluorescence images, and the video frame with the largest S value is selected as the optimal image, which is then fused with the fluorescence image acquired later in the two fluorescence images (i.e., the latter fluorescence image).

[0057] All pixels in the optimal image and the fluorescence image are converted to the same coordinate system (i.e., the coordinate axes of the optimal frame of the video image and the fluorescence image are unified). Specifically: for the video image and the fluorescence image to be fused, a two-dimensional coordinate system for each of the two images is generated according to the three calibration points in the image (i.e., based on the calibration points, the optimal image and the fluorescence image are converted into the two-dimensional coordinate system respectively); and the two two-dimensional coordinate systems are calibrated (unified) using the relative position deviation value (a, b) determined previously (i.e., the coordinate difference of in vitro calibration), so that each pixel of the two images is in the same two-dimensional coordinate system.

[0058] Step 4: For the position with only a single pixel in the coordinate system, clear the pixel value to zero and only keep the pixel value of the overlapping pixel. That is, determine whether the pixel value in the coordinate is an odd number. If so, delete the pixel value; otherwise, go to step 5.

[0059] Step 5: Average the pixel values ​​of the two pixels, and then save the average value to form a fused image P, such as Figure 5 That is, zi=(xi+yi) / 2, where xi represents the pixel value of the i-th pixel in the optimal image, yi represents the pixel value of the i-th pixel in the fluorescence image, and zi represents the pixel value of the i-th pixel in the fused image P.

[0060] In the present invention, after obtaining the fused single image, the image needs to be reconstructed to form a video file, thereby completing the fusion of the video image and the fluorescent image.

[0061] Step 6: Arrange multiple fused images (fused images) P1\P2\P3... in chronological order, and export the images at a setting of 30 frames per second, thereby completing the fusion of the video image and the fluorescent image. Wherein, Pi represents the i-th fused image.

[0062] This embodiment provides a method for fusing fluorescence images and video images, which selects the optimal frame video image to fuse with the fluorescence image, and then reorders them to form a video image, thereby completing the precise fusion of the video image and the fluorescence image, so that the patient's fluorescence can be collected in real time during the use of the endoscope.

[0063] The present embodiment provides a method for fusing fluorescence images and video images, which fuses video images with fluorescence images and can display fluorescence images in real time in the video, breaking through the limitation of single-frame fluorescence imaging, allowing doctors to continuously observe the fluorescence imaging conditions in the patient's body to assist the doctor's diagnosis.

[0064] The present embodiment provides a method for fusing fluorescence images and video images, which fuses fluorescence images and video images containing more data information, so that more biochemical information in the patient's body can be accurately displayed in real time when an endoscope is used for examination, which helps doctors make diagnoses, improves diagnostic efficiency, and alleviates patients' pain.

[0065] Embodiment 2

[0066] This embodiment provides a system for fusing fluorescence images and video images.

[0067] A fluorescence image and video image fusion system, comprising:

[0068] A data acquisition module is configured to: acquire a plurality of frames of continuous video images and a plurality of continuous fluorescence images within a period of time;

[0069] The image selection module is configured to: for all video images between two adjacent fluorescent images, respectively calculate the focus value of the reference column of each frame of the video image, and select the video image with the largest focus value as the optimal image; the focus value calculation step includes: for the reference column of the video image, calculating the difference in grayscale values ​​between two adjacent pixel points; and taking the sum of all the differences as the focus value;

[0070] The image fusion module is configured to: after converting all pixels in the optimal image and the fluorescent image to the same coordinate system, fuse the optimal image and the fluorescent image to obtain a fused image; the fusion method is: calculate the average value of the pixel values ​​of the pixel points in the optimal image and the fluorescent image, and use the average value as the pixel value of the pixel points in the fused image.

[0071] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0072] Embodiment 3

[0073] This embodiment provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor. When the program is executed by the processor, the steps in the method for generating three-dimensional artistic characters as described in the first embodiment are implemented.

[0074] Embodiment 4

[0075] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps in the method for generating three-dimensional artistic characters as described in the first embodiment are implemented.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0077] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for fusing a fluorescence image and a video image, characterized in that: include: Acquire a plurality of frames of continuous video images and a plurality of continuous fluorescence images within a period of time; For all video images between two adjacent fluorescent images, the focus value of the reference column of each frame of the video image is calculated respectively, and the video image with the largest focus value is selected as the optimal image; the step of calculating the focus value includes: for the reference column of the video image, calculating the difference in grayscale values ​​between two adjacent pixel points; and taking the sum of all the differences as the focus value; The reference column is a fixed column specified artificially, or the reference column is a column with the smallest sum of variances, and the sum of variances is: D i =∑ i d ij , where D i represents the sum of variances of the i-th column, N represents the number of pixels in the i-th column of each j-frame video image between two fluorescent images, represents the mean gray value of all pixels in the i-th column of each j-frame video image between two fluorescent images, x ijk represents the gray value of the kth pixel in the i-th column of each j-frame video image between two fluorescent images; After all pixels in the optimal image and the fluorescent image are converted to the same coordinate system, the optimal image and the fluorescent image are fused to obtain a fused image; the fusion method is: calculating the average value of the pixel values ​​of the pixels in the optimal image and the fluorescent image, and taking the average value as the pixel value of the pixel in the fused image; and arranging all the fused images in chronological order, exporting the images according to the set number of frames, and completing the fusion of the video image and the fluorescent image.

2. The method for fusing a fluorescence image and a video image according to claim 1, characterized in that: For the optimal image and the fluorescence image converted to the same coordinate system, the pixel values ​​of the positions with only a single pixel in the coordinate system are cleared to zero, and then the optimal image and the fluorescence image are fused.

3. The method for fusing a fluorescence image and a video image according to claim 1, characterized in that: The step of converting all pixels in the optimal image and the fluorescence image to the same coordinate system includes: based on multiple calibration points, converting the optimal image and the fluorescence image into two-dimensional coordinate systems respectively; and converting the pixels in the optimal image and the fluorescence image into the same two-dimensional coordinate system through relative position deviation.

4. A system for fusing fluorescence images and video images, characterized in that: include: A data acquisition module is configured to: acquire a plurality of frames of continuous video images and a plurality of continuous fluorescence images within a period of time; The image selection module is configured to: for all video images between two adjacent fluorescent images, respectively calculate the focus value of the reference column of each frame of the video image, and select the video image with the largest focus value as the optimal image; the reference column is a fixed column specified manually, or the reference column is a column with the smallest variance sum, and the variance sum is: D i =∑ i d ij , where D i represents the sum of variances of the i-th column, N represents the number of pixels in the i-th column of each j-frame video image between two fluorescent images, represents the mean gray value of all pixels in the i-th column of each j-frame video image between two fluorescent images, x ijk represents the gray value of the kth pixel in the i-th column of each j-frame video image between two fluorescent images; The image fusion module is configured to: after converting all pixels in the optimal image and the fluorescent image to the same coordinate system, fuse the optimal image and the fluorescent image to obtain a fused image; the fusion method is: calculate the average value of the pixel values ​​of the pixels in the optimal image and the fluorescent image, and use the average value as the pixel value of the pixel in the fused image; and arrange all the fused images in chronological order, export the images according to the set number of frames, and complete the fusion of the video image and the fluorescent image.

5. The system for fusing fluorescence images and video images according to claim 4, characterized in that: For the optimal image and the fluorescence image converted to the same coordinate system, the pixel values ​​of the positions with only a single pixel in the coordinate system are cleared to zero, and then the optimal image and the fluorescence image are fused.

Images