Image processing method, device, system and electronic equipment

By using multi-channel cameras and image processing technology, multiple fluorescence images can be simultaneously imaged in visible light, solving the problem of imaging channel limitations in conventional systems and improving the accuracy of lesion localization and the convenience of observation.

CN118476789BActive Publication Date: 2024-11-26NANJING NUOYUAN MEDICAL DEVICES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410947516.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-26
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Conventional fluorescence surgical navigation systems can only perform dual-channel imaging, and cannot simultaneously use multiple probes to image the lesion area. Furthermore, the fusion algorithm is ineffective and obscures the details of the lesion area in the visible light image.

Method used

Visible light images and multiple fluorescence images are acquired by a multi-channel camera. Channel decomposition and image fusion are performed to generate a multi-image stitched image. Image enhancement algorithms and pyramid decomposition methods are used to improve image quality.

Benefits of technology

It enables simultaneous imaging of different fluorescence images in visible light images, improving the accuracy of lesion localization and assisting doctors in accurate analysis and judgment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118476789B_ABST
    Figure CN118476789B_ABST
Patent Text Reader

Abstract

The present invention provides an image processing method, device, system and electronic equipment, which relates to the field of medical image processing technology, including: obtaining an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; the image to be processed includes a visible light image and multiple fluorescence images; performing channel decomposition on the visible light image to obtain multiple channel images; performing image fusion on each fluorescence image and the channel image to obtain multiple fused images; and generating a multi-screen spliced ​​image of the target object based on the multiple fused images. The present invention can simultaneously image different fluorescence images in a visible light image, improve the convenience of observation, and improve the accuracy of lesion positioning, assisting doctors in more accurate analysis and judgment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image processing technology, and in particular to an image processing method, device, system and electronic equipment. Background Art

[0002] With the rapid development of modern science and technology, especially the increasing maturity of imaging technology and fluorescent molecular probes, fluorescent surgical navigation systems have also made great progress. Fluorescent molecular imaging, endoscopic imaging and other methods have greatly improved the surgical efficiency and quality of tumor surgeries. The three-channel camera can simultaneously capture visible light images and two specific bands of fluorescent images. Combined with the corresponding fluorescent probes, it can intuitively reflect the lesion area through the fluorescent position, and dual-fluorescence band imaging allows doctors to use two disease-related probes for imaging, which can further improve the imaging effect of the lesion area, thereby providing guidance for doctors' diagnosis and surgery. Therefore, it plays an important role in the different stages of early tumor diagnosis, mid-term treatment and late observation.

[0003] Conventional fluorescence surgical navigation systems can only perform dual-channel imaging, i.e., visible light channel + fluorescence channel. The limitation of imaging channels means that users can only observe a certain type of fluorescence and cannot use multiple probes to image the lesion area at the same time. In addition, conventional fusion algorithms simply superimpose the fluorescence image on a certain channel of the visible light image at a certain ratio. Therefore, the display effect after fusion is poor, and it may even cover up the details of the lesion area that originally existed in the visible light image. The above problems are common problems faced by conventional dual-channel camera imaging devices. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an image processing method, device, system and electronic equipment, which can enable different fluorescent images to be imaged simultaneously in a visible light image, thereby improving the convenience of observation and the accuracy of lesion localization, and assisting doctors in making more accurate analysis and judgment.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising: acquiring an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; the image to be processed comprises a visible light image and a plurality of fluorescence images; performing channel decomposition on the visible light image to obtain a plurality of channel images; performing image fusion on each fluorescence image with the channel image to obtain a plurality of fused images; and generating a multi-screen stitched image of the target object based on the plurality of fused images.

[0006] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the step of fusing each fluorescence image with a channel image to obtain multiple fused images includes: selecting a target channel image from multiple channel images of the visible light image, and performing image decomposition on the target channel image to obtain multiple layers of first decomposed images; performing image decomposition on each fluorescence image to obtain multiple layers of second decomposed images corresponding to each fluorescence image; performing dual-channel fusion of each second decomposed image with the first decomposed image to obtain a first fused image corresponding to each fluorescence image; performing multi-channel fusion of each second decomposed image with the first decomposed image to obtain a second fused image containing multiple fluorescence images; and generating multiple fused images based on the first fused image and the second fused image.

[0007] In combination with the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the step of performing dual-channel fusion of each second decomposition image with the first decomposition image to obtain a first fused image corresponding to each fluorescence image includes: superimposing the second decomposition image of the fluorescence image with the first decomposition image in a preset ratio to obtain a fused decomposition image; reconstructing the fused decomposition image to obtain a channel reconstructed image that incorporates fluorescence information; and merging the channel reconstructed image with the visible light image to obtain a first fused image corresponding to the current fluorescence image.

[0008] In combination with the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein there are multiple target channel images; each second decomposed image is multi-channel fused with the first decomposed image to obtain a second fused image containing multiple fluorescence images, comprising: channel fusion of each second decomposed image with the first decomposed image of the corresponding target channel image to obtain a fused channel image corresponding to each target channel image; the corresponding fluorescence image is fused into each fused channel image; and the fused channel image is merged with the visible light image to obtain a second fused image.

[0009] In combination with the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the above method further includes: performing image processing on the visible light image by using a preset image enhancement algorithm.

[0010] In combination with the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the above method also includes: synchronizing the visible light image and the fluorescent image to obtain a group of images collected at the same time; performing image preprocessing on the image group; the image preprocessing includes color correction, brightness correction, and image denoising.

[0011] In combination with the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the step of generating a multi-screen stitched image of a target object based on multiple fused images includes: grouping the multiple fused images into a multi-mode image group; performing image selection on the fused images in the multi-mode image group to obtain a target image; and performing image stitching on the target images to generate a multi-screen stitched image of the target object.

[0012] In a second aspect, an embodiment of the present invention further provides an image processing device, wherein the device includes: a data acquisition module, used to acquire an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; the image to be processed includes a visible light image and multiple fluorescence images; a processing module, used to perform channel decomposition on the visible light image to obtain multiple channel images; a fusion module, used to perform image fusion on each fluorescence image with the channel image to obtain multiple fused images; an image generation module, used to generate a multi-screen stitched image of the target object based on the multiple fused images.

[0013] In a third aspect, an embodiment of the present invention further provides an image processing system, wherein the image processing system is configured with the above-mentioned device for executing the above-mentioned method.

[0014] In a fourth aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the above method is implemented when the processor executes the computer program.

[0015] The embodiments of the present invention bring the following beneficial effects: an image processing method, device, system and electronic device provided by the present invention perform channel decomposition on a visible light image to obtain multiple channel images, and then fuse each acquired fluorescence image with the channel image of the visible light image to obtain multiple fused images. These fused images enable different fluorescence images to be imaged simultaneously in the visible light image, thereby improving the convenience of observation and the accuracy of lesion localization, and assisting doctors in making more accurate analysis and judgments.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of another image processing method provided by an embodiment of the present invention;

[0021] Figure 3 An image processing flow chart provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the structure of another image processing device provided by an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0026] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described in the present invention may be embodied in a wide variety of forms, and any specific structure and / or function described in the present invention is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described in the present invention may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described in the present invention may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described in the present invention may be used to implement this device and / or practice this method.

[0027] It should also be noted that the diagrams provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure, and the diagrams only show the components related to the present disclosure rather than the number, shape and size of the components in actual implementation. The type, quantity and proportion of each component in actual implementation can be changed at will, and the component layout type may also be more complicated. In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that aspects can be practiced without these specific details.

[0028] The embodiments of the present invention provide an image processing method, device, system and electronic device, which can enable different fluorescent images to be imaged simultaneously in a visible light image, thereby improving the convenience of observation and the accuracy of lesion positioning, and assisting doctors in making more accurate analysis and judgment.

[0029] To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present invention is first introduced in detail. Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention is shown. Figure 1 As shown, the method comprises the following specific steps:

[0030] Step S101, obtaining an image to be processed.

[0031] Specifically, the image to be processed in the embodiment of the present invention is obtained by photographing the target object with a medical multi-channel camera; the image to be processed includes a visible light image and a plurality of fluorescent images. In one embodiment, a three-channel camera is used to capture visible light images A, fluorescent images B and fluorescent images C in the field of view, and are converted into visible light images A, fluorescent images B and fluorescent images C through internal processing of the camera. During the acquisition process, the imaging effect can be adjusted by camera parameters, and the adjustable parameters include but are not limited to independent pixel formats, exposure time, pixel gain, black level range, etc. of the three channels. The acquired three-channel image is transmitted by the camera SDK to the processing software for subsequent processing.

[0032] Among them, the angles of acquisition of visible light images and fluorescent images are exactly the same, the difference is that each fluorescent image can use different fluorescent probes and filters to capture images of specific wavelengths. For example, the first fluorescent channel of a medical multi-channel camera can locate a tumor by collecting the excitation light of the first probe, and the second fluorescent channel can locate a cyst by collecting the excitation light of the second probe. Furthermore, these images are fused by the fusion method of an embodiment of the present invention to fuse different fluorescent images into visible light images, so that multiple disease information can be observed simultaneously on one image.

[0033] Step S102, performing channel decomposition on the visible light image to obtain a plurality of channel images.

[0034] Step S103: fusing each fluorescence image with the channel image to obtain a plurality of fused images.

[0035] In a specific implementation, the embodiment of the present invention performs channel decomposition on the visible light image to obtain RGB three-channel images R, G and B. Further, each fluorescence image and the visible light image in a specific channel are fused to highlight the corresponding fluorescence information in the specific channel of the image. The final fused image combines the low-frequency features and high-frequency details in the fluorescence image and highlights them with a specific color, which is convenient for researchers to observe the visible light information and the fluorescence image feature information in the fused image at the same time.

[0036] Step S104: generating a multi-image stitching image of the target object based on the multiple fused images.

[0037] The multiple fused images can respectively display different fluorescent image details in the visible light image. Furthermore, the embodiment of the present invention generates a multi-screen stitched image corresponding to the target object based on the multiple fused images to display the target object. In one embodiment, the selection and switching of modes can be realized by selecting and processing the multiple fused images, and the images are stitched based on the selection results to obtain a multi-screen stitched image for display.

[0038] An image processing method provided by an embodiment of the present invention performs channel decomposition on a visible light image to obtain multiple channel images, and then fuses each acquired fluorescence image with the channel image of the visible light image to obtain multiple fused images. These fused images enable different fluorescence images to be imaged simultaneously in the visible light image, thereby improving the convenience of observation and the accuracy of lesion localization, and assisting doctors in making more accurate analysis and judgments.

[0039] Furthermore, in the imaging scenario using a medical endoscope, since the endoscope has a small field of view and needs to penetrate into the body, it relies on a point light source as the main light source of the visible light image. Therefore, there are inevitably problems such as uneven illumination of the image, severe tissue occlusion, large brightness difference between foreground and background, overexposure or underexposure. Based on the above embodiment, the embodiment of the present invention also provides another image processing method, Figure 2 FIG. 4 is a flowchart of another image processing method provided by an embodiment of the present invention. Figure 2 As shown, the following steps are included:

[0040] Step S201, obtaining an image to be processed.

[0041] Step S202, performing channel decomposition on the visible light image to obtain multiple channel images.

[0042] On the basis of the above embodiment, in order to eliminate the time difference between the images to be processed, the embodiment of the present invention further performs image synchronization on the visible light image and the fluorescence image to obtain an image group collected at the same time.

[0043] Furthermore, the embodiment of the present invention also performs image preprocessing on the visible light image and the fluorescent image of the image group respectively, and the image preprocessing includes but is not limited to color correction, brightness correction, image denoising, etc. It should be noted that the present invention does not limit the specific implementation method of the image processing algorithm used in each step of the image processing flow, such as image denoising can be implemented using mean filtering, Gaussian filtering, or other denoising methods.

[0044] In particular, for visible light images, the embodiment of the present invention also processes the visible light images through a preset image enhancement algorithm to enhance the display effect of the visible light images. In one embodiment, the embodiment of the present invention uses a multi-scale retina algorithm with color restoration (Multi-Scale Retinex with Color Restoration, MSRCR) to process to enhance its display effect. This method uses the color restoration factor C to adjust the proportional relationship between the three color channels in the original image, thereby highlighting the information of the relatively dark area, thereby eliminating the defect of image color distortion. Finally, a preprocessed image group is obtained.

[0045] Further, the image group is fused by the following steps. In one embodiment, the present invention processes the visible light image and the fluorescence image of the image to be processed by using an image fusion method based on MSRCR and image pyramid to obtain a fused image with enhanced high-frequency details. The image fusion step of the present invention is referred to the following steps.

[0046] Step S203 , selecting a target channel image from the multiple channel images of the visible light image, and performing image decomposition on the target channel image to obtain a multi-layer first decomposed image.

[0047] In a specific implementation, the embodiment of the present invention selects a target channel image for image fusion, such as a G channel image, from the multiple channel images R, G, and B after the visible light image is decomposed.

[0048] Further, the image is decomposed using a preset decomposition method to obtain a multi-layer first decomposed image. The embodiment of the present invention performs Laplacian pyramid decomposition on the target channel image, and correspondingly obtains a multi-layer Laplacian pyramid image. Laplacian pyramid decomposition is mainly used for multi-scale expression and reconstruction of images, especially in image scaling and reconstruction. It is based on each layer of the image of the Gaussian pyramid, and obtains a series of difference images by subtracting the predicted image after upsampling and Gaussian convolution of the previous layer of the image, and these difference images are the decomposed images of the Laplacian pyramid. This process allows the image to be restored to the greatest extent, especially when a small image is reconstructed into a large image.

[0049] Step S204 , performing image decomposition on each fluorescent image to obtain a multi-layer second decomposed image corresponding to each fluorescent image.

[0050] Referring to the above steps, after determining the target channel image and performing image decomposition on it, the embodiment of the present invention further performs image decomposition on each fluorescence image to obtain a multi-layer second decomposition image corresponding to each fluorescence image. In particular, each fluorescence image can also be subjected to pyramid decomposition to obtain a Laplacian pyramid that preserves high-frequency information such as texture boundaries.

[0051] Among them, the embodiment of the present invention does not limit the color channel (that is, the target channel image) of the color image fusion process. For example, the fluorescence image can be fused to the RGB channel of the color image, or decomposed and fused in other ways, such as converting to the HSV color space for fusion.

[0052] Step S205 , dual-channel fusion is performed on each second decomposed image and the first decomposed image to obtain a first fused image corresponding to each fluorescent image.

[0053] Step S206: Perform multi-channel fusion on each second decomposed image and the first decomposed image to obtain a second fused image including a plurality of fluorescence images.

[0054] When fusing a fluorescence image and a visible light image, the embodiment of the present invention designs a dual-channel image fusion method and a multi-channel image fusion method to respectively superimpose different fluorescence images and specific channels of the visible light image to obtain multiple different fused images. Figure 3The image processing flow chart corresponding to the embodiment of the present invention is shown to illustrate the image processing process and fusion steps. Among them, the fused image obtained by the embodiment of the present invention can simultaneously fuse the low-frequency features and high-frequency details in different fluorescent images into the visible light image, and express them in different colors. In the final fused image, researchers can simultaneously observe the visible light information and the feature information contained in multiple fluorescent images, which is convenient for analysis.

[0055] In a specific implementation, for the dual-channel fusion method, the embodiment of the present invention obtains a fused decomposed image by superimposing the second decomposed image of the fluorescence image with the first decomposed image according to a preset ratio. Further, the fused decomposed image is reconstructed to obtain a channel reconstructed image that incorporates fluorescence information. Thereafter, the channel reconstructed image is merged with the visible light image to obtain a first fused image corresponding to the current fluorescence image.

[0056] Specifically, the Laplacian pyramid image of the corresponding fluorescence image may be superimposed with the Laplacian pyramid image of the to-be-fused channel of the visible light image in a certain ratio to obtain a fused Laplacian pyramid.

[0057] Furthermore, the fused Laplacian pyramid image is reconstructed to obtain a channel image to be fused (i.e., a target channel image) fused with fluorescence information. Furthermore, the channel image to be fused is merged with other channel images obtained after channel decomposition to obtain a fused image fused with fluorescence image information in a specific channel. The fused image is a first fused image, including a fused image A and a fused image B with enhanced high-frequency details, each corresponding to a fluorescence image.

[0058] Furthermore, for the multi-channel fusion method, there are multiple target channel images, that is, multiple channel images can be selected as target channel images, such as a G channel image and a B channel image. Correspondingly, the first decomposed image includes a decomposed image of the G channel image (such as LA_G) and a decomposed image of the B channel image (such as LA_B). Further, each second decomposed image is channel-fused with the first decomposed image of the corresponding target channel image to obtain a fused channel image corresponding to each target channel image. For example, the second decomposed image LB of one fluorescence image is superimposed with LA_G to obtain LA_G'; the second decomposed image LC of another fluorescence image is superimposed with LA_B to obtain LA_B'. LA_G' and LA_B' are the corresponding fused channel images, respectively. The corresponding fluorescence image is fused in each fused channel image.

[0059] Among them, in the fusion step of the three-channel fusion method, similar to the two-channel fusion method, the second decomposition image (such as the Laplacian pyramid image) of the fluorescence image is superimposed with the first decomposition image (such as the Laplacian pyramid image) of the target fusion channel of the visible light image according to a certain ratio to obtain the fused decomposition image. Corresponding to the Laplacian pyramid image, it is the Laplacian pyramid. Further, the fused Laplacian pyramid images LA_G' and LA_B' are reconstructed respectively to obtain the channel images FA and FB to be fused with the fluorescence information.

[0060] Furthermore, the channel images FA and FB to be fused are combined with other channel images R of the visible light image after the channel decomposition, to obtain a fused image that fuses the information of the fluorescence image B and the fluorescence image C in a specific channel. This fused image is the second fused image.

[0061] Step S207: generating a plurality of fused images based on the first fused image and the second fused image.

[0062] After the above steps, visible light image A, fluorescence image B, fluorescence image C, ..., fused image A+B, fused image A+C and fused image A+B+C are obtained, and the number of fused images is determined based on the number of fluorescence images. Among them, all the above images can form a multi-mode image group.

[0063] Step S208: generating a multi-image stitching image of the target object based on the multiple fused images.

[0064] In a specific implementation, the embodiment of the present invention selects the fused image in the multi-mode image group to obtain the target image, and realizes the selection and switching of the mode. The target image is stitched based on the selection result to generate a multi-image stitching image of the target object, which can display the image of the target object.

[0065] Another image processing method provided by an embodiment of the present invention, based on adding a fluorescent imaging channel to realize simultaneous imaging of multiple probes, adopts a certain method to realize the enhancement of visible light images, so as to ensure its imaging effect in a specific usage scenario. At the same time, by using a dual-channel image fusion and a three-channel image fusion method designed for medical images, the different fluorescent channel information of the three-channel camera is simultaneously fused into the visible light image, which improves the convenience of observation and the accuracy of lesion positioning, and assists doctors in more accurate analysis and judgment.

[0066] Among them, the embodiment of the present invention proposes a dual-channel image fusion method and a multi-channel image fusion method based on MSRCR and image pyramid. On the basis of using MSRCR to enhance the display effect of visible light images, the visible light image and the fluorescent image are respectively decomposed into pyramids to obtain a Laplacian pyramid that preserves high-frequency information such as texture boundaries. And through pyramid inverse transformation and channel merging, the Laplacian pyramid of the fluorescent image is fused to a specific channel of the visible light image at a certain ratio. The final fused image integrates the low-frequency features and high-frequency details in the fluorescent image, and is highlighted with a specific color, which is convenient for researchers to observe the visible light information and the fluorescent image feature information in the fused image at the same time, and is convenient for analysis.

[0067] In addition, the embodiments of the present invention can realize various image processing functions including color correction, brightness correction, image denoising, image enhancement, image fusion, image stitching, image switching, etc. by collecting and processing visible light images and fluorescence images captured by a multi-channel camera. At the same time, camera parameters such as gain, exposure time, etc. can be adjusted based on the camera SDK to achieve rich imaging effects to meet user needs.

[0068] Furthermore, a multi-scale retinal algorithm with color restoration is used to enhance color images, which can effectively reduce the adverse effects of poor medical imaging conditions, especially endoscopic imaging conditions, such as tissue occlusion causing the image background to be too dark, and uneven light source causing low brightness in local areas.

[0069] Based on the above system embodiment, the embodiment of the present invention further provides an image processing device, which is applied to the above method. Figure 4 FIG. 4 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. Figure 4 As shown, the device includes: a data acquisition module 100, used to acquire an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; the image to be processed includes a visible light image and a plurality of fluorescence images; a processing module 200, used to perform channel decomposition on the visible light image to obtain a plurality of channel images; a fusion module 300, used to perform image fusion on each fluorescence image with the channel image to obtain a plurality of fused images; an image generation module 400, used to generate a multi-screen spliced ​​image of the target object based on the plurality of fused images.

[0070] The image processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned image processing method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned image processing method embodiment.

[0071] Further, in Figure 4 On the basis of Figure 5A schematic diagram of the structure of another image processing device provided by an embodiment of the present invention is also shown. Figure 5 The fusion module 300 shown is also used to select a target channel image from multiple channel images of the visible light image, perform image decomposition on the target channel image to obtain a multi-layer first decomposition image; perform image decomposition on each fluorescence image to obtain a multi-layer second decomposition image corresponding to each fluorescence image; perform dual-channel fusion of each second decomposition image with the first decomposition image to obtain a first fused image corresponding to each fluorescence image; perform multi-channel fusion of each second decomposition image with the first decomposition image to obtain a second fused image containing multiple fluorescence images; and generate multiple fused images based on the first fused image and the second fused image.

[0072] Furthermore, the above-mentioned fusion module 300 is also used to superimpose the second decomposition image of the fluorescence image with the first decomposition image according to a preset ratio to obtain a fused decomposition image; reconstruct the fused decomposition image to obtain a channel reconstructed image that integrates the fluorescence information; and merge the channel reconstructed image with the visible light image to obtain a first fused image corresponding to the current fluorescence image.

[0073] Furthermore, there are multiple target channel images; the above-mentioned fusion module 300 is also used to perform channel fusion on each second decomposed image with the first decomposed image of the corresponding target channel image to obtain a fused channel image corresponding to each target channel image; each fused channel image is fused with the corresponding fluorescence image; the fused channel image is merged with the visible light image to obtain a second fused image.

[0074] Furthermore, except Figure 4 In addition to the structure shown, Figure 5 The system also includes a preprocessing module 500, which is used to perform image processing on the visible light image by using a preset image enhancement algorithm.

[0075] Furthermore, the above-mentioned preprocessing module 500 is also used to synchronize the visible light image and the fluorescent image to obtain an image group collected at the same time; and perform image preprocessing on the image group, and the image preprocessing includes color correction, brightness correction, and image denoising.

[0076] Furthermore, the above-mentioned image generation module 400 is also used to form a multi-mode image group with multiple fused images; perform image selection on the fused images in the multi-mode image group to obtain a target image; and perform image stitching on the target image to generate a multi-screen stitched image of the target object.

[0077] Furthermore, an embodiment of the present invention also provides an image processing system, which is configured with the above-mentioned device and is used to execute Figures 1 to 3Any of the methods shown. In one embodiment, the method comprises three modules: a multi-channel camera, a core processing module and a display. The multi-channel camera is used to shoot the target object to obtain a visible light image and a fluorescent image. The core processing module is a software processing flow, which consists of four parts: image acquisition, image preprocessing, image fusion, and image display. The core processing module is used to process the image, obtain the fused image and the multi-image spliced ​​image, and then display the image on the display.

[0078] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, Figures 1 to 3 The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor when it is run. Figures 1 to 3 Any of the steps of the method shown.

[0079] The embodiment of the present invention also provides a schematic diagram of the structure of an electronic device, such as Figure 6 FIG. 1 is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer executable instructions that can be executed by the processor 61, and the processor 61 executes the computer executable instructions to implement Figures 1 to 3 Any of the methods shown.

[0080] exist Figure 6In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63, wherein the processor 61, the communication interface 63 and the memory 60 are connected via the bus 62. The memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. may be used. The bus 62 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. It may also be an AMBA (Advanced Microcontroller Bus Architecture) bus, where AMBA defines three types of buses, including an APB (Advanced Peripheral Bus) bus, an AHB (Advanced High-performance Bus) bus, and an AXI (Advanced Xtensible Interface) bus. The bus 62 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 61 or the instructions in the form of software. The above processor 61 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the above-mentioned in combination with its hardware. Figures 1 to 3 Any of the methods shown.

[0082] The computer program product of an image processing method, device, system and electronic device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can refer to the method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the previous method embodiment, which will not be repeated here.

[0083] It should be noted that, based on the image processing flow framework used in the present invention, the method of adding or reducing the function in a certain step but not changing the flow framework structure and using the same control method as the present invention is also within the scope of the present invention. If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes. In the description of the present invention, it should be noted that the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0084] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image processing method, It is characterized in that include: Acquire an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; The image to be processed includes a visible light image and a plurality of fluorescence images; Performing channel decomposition on the visible light image to obtain a plurality of channel images; Performing image fusion on each of the fluorescence images and the channel images to obtain a plurality of fused images; Generate a multi-image stitching image of the target object based on the multiple fused images; The step of fusing each of the fluorescence images with the channel images to obtain a plurality of fused images comprises: Selecting a target channel image from a plurality of channel images of the visible light image, and performing image decomposition on the target channel image to obtain a multi-layer first decomposed image; Decomposing each of the fluorescent images to obtain a multi-layer second decomposed image corresponding to each fluorescent image; Performing dual-channel fusion on each of the second decomposed images and the first decomposed image to obtain a first fused image corresponding to each fluorescent image; Perform multi-channel fusion of each of the second decomposed images with the first decomposed image to obtain a second fused image including a plurality of fluorescence images; A plurality of fused images are generated based on the first fused image and the second fused image.

2. The method according to claim 1, It is characterized in that The step of performing dual-channel fusion of each of the second decomposed images with the first decomposed image to obtain a first fused image corresponding to each fluorescent image comprises: superimposing the second decomposed image of the fluorescence image with the first decomposed image according to a preset ratio to obtain a fused decomposed image; Reconstructing the fused decomposed image to obtain a channel reconstructed image fused with fluorescence information; The channel reconstructed image is combined with the visible light image to obtain a first fused image corresponding to the current fluorescence image.

3. The method according to claim 2, It is characterized in that There are multiple target channel images; and the step of performing multi-channel fusion of each of the second decomposed images with the first decomposed image to obtain a second fused image containing multiple fluorescence images includes: Perform channel fusion on each of the second decomposed images and the first decomposed image of the corresponding target channel image to obtain a fused channel image corresponding to each of the target channel images; each of the fused channel images is fused with a corresponding fluorescent image; The fused channel image is combined with the visible light image to obtain a second fused image.

4. The method according to claim 1, It is characterized in that The method further comprises: The visible light image is processed by a preset image enhancement algorithm.

5. The method according to claim 1, It is characterized in that The method further comprises: Performing image synchronization on the visible light image and the fluorescent image to obtain an image group acquired at the same time; The image group is subjected to image preprocessing; the image preprocessing includes color correction, brightness correction, and image denoising.

6. The method according to claim 1, It is characterized in that The step of generating a multi-image stitching image of the target object based on the multiple fused images comprises: Combining a plurality of the fused images into a multi-modal image group; Performing image selection on the fused image in the multi-modal image group to obtain a target image; The target images are stitched together to generate a multi-image stitched image of the target object.

7. An image processing device, It is characterized in that The device comprises: A data acquisition module is used to acquire an image to be processed; the image to be processed is obtained by photographing a target object with a medical multi-channel camera; the image to be processed includes a visible light image and a plurality of fluorescence images; A processing module, used for performing channel decomposition on the visible light image to obtain multiple channel images; A fusion module, used for fusing each of the fluorescence images with the channel images to obtain a plurality of fused images; An image generation module, used for generating a multi-image stitching image of the target object based on the multiple fused images; The fusion module is further used to select a target channel image from the multiple channel images of the visible light image, perform image decomposition on the target channel image to obtain a multi-layer first decomposition image; perform image decomposition on each of the fluorescence images to obtain a multi-layer second decomposition image corresponding to each fluorescence image; perform dual-channel fusion on each of the second decomposition images with the first decomposition image to obtain a first fused image corresponding to each fluorescence image; perform multi-channel fusion on each of the second decomposition images with the first decomposition image to obtain a second fused image containing multiple fluorescence images; and generate multiple fused images based on the first fused image and the second fused image.

8. An image processing system, It is characterized in that The image processing system is configured with the apparatus of claim 7, and is used to execute the method of any one of claims 1-6.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor. It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.