A multi-modal imaging fusion system, device, and method

Abstract

The application is suitable for the field of imaging processing technology, and provides a multi-mode imaging fusion system, device and method. The system comprises: an external interaction unit for acquiring an imaging fusion mode; an image source unit for acquiring the imaging fusion mode through the fusion processing unit, adjusting an imaging focus position based on the imaging fusion mode, generating a plurality of imaging images based on the imaging focus position, and feeding back the imaging images to the fusion processing unit; the fusion processing unit is used for respectively extracting image information from a plurality of imaging images to generate a fusion data set, adopting a streaming fusion method to perform fusion processing on the fusion data set, and obtaining a fusion result image; and an image output unit is used for visualizing the fusion result image. The multi-mode imaging fusion system of the application has a real-time fusion method and a fusion implementation method suitable for various scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of imaging processing, and particularly relates to a multi-mode imaging fusion system, device and method. BACKGROUND

[0002] The imaging processing system is generally applied to multiple fields such as optics, teaching, research and civil use, and is generally connected, embedded or split in various optical system devices, including but not limited to embedded body type microscopic cameras, embedded biological microscopic cameras, embedded split light microscopes, cameras connected to the optical path of the microscope eyepiece end, split cameras for the optical path of the microscope objective end, and electronic eyepieces at the end of a telescope.

[0003] The imaging processing system applied to the above optical system devices in the prior art mostly only has a basic camera attribute adjustment function, and in addition, for imaging devices with an imaging focus position adjustment function, most of them only realize single control of the focus distance, and cannot provide a composite image containing multiple focus position images, thereby limiting the diversity of imaging and the acquisition of depth information, and further, in application scenarios requiring real-time acquisition and processing of multiple focus position images, the above imaging processing system is not sufficient, and cannot provide sufficient flexibility and real-time performance.

[0004] Therefore, the present application provides a multi-mode imaging fusion system with real-time performance and different fusion implementation mechanisms for various scenes. SUMMARY

[0005] The embodiments of the present application provide a multi-mode imaging fusion system, device and method, which can solve one of the above-mentioned problems in the prior art.

[0006] A multi-mode imaging fusion system, comprising: an image source unit, a fusion processing unit, an external interaction unit and an image output unit;

[0007] The external interaction unit is configured to obtain an imaging fusion mode and feed back the imaging fusion mode to the fusion processing unit;

[0008] The image source unit is configured to obtain the imaging fusion mode through the fusion processing unit, adjust an imaging focus position based on the imaging fusion mode, generate a plurality of imaging images based on the imaging focus position, and feed back the imaging images to the fusion processing unit;

[0009] The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, extract image information from multiple imaging images to generate a fusion data set, perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feed back the fusion result image to the image output unit.

[0010] The image output unit is used to receive the fused result image and visualize the fused result image.

[0011] Furthermore, the fusion processing unit includes a fusion logic module, an image cache module, and a fusion processing module;

[0012] The fusion logic module is used to receive and feed back the interaction instructions from the external interaction unit to the image cache module and the image source unit;

[0013] The image caching module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module;

[0014] The fusion processing module is used to receive the fusion data set, and to perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and to feed back the fusion result image to the image output unit and the external interaction unit.

[0015] Furthermore, the image caching module includes a data receiving submodule, a pre-preparation submodule, and a data collection submodule, and the image data packet includes an image, an image grid matrix, and a second pyramid;

[0016] The data receiving submodule is used to receive source images and imaging images;

[0017] The pre-preparation submodule is used to extract image information from multiple imaging images to generate image data packets;

[0018] The data set submodule is used to generate a fused data set based on multiple image data packets.

[0019] Furthermore, the step of extracting image information from each of the multiple imaging images to generate an image data packet includes:

[0020] An average gradient grid matrix is ​​constructed based on the preset grid specifications of the grid cells;

[0021] If the image is a color image, then the image is converted into a grayscale image;

[0022] The grayscale image or the imaging image is preprocessed, and a gradient operator is used to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaging image.

[0023] Calculate the average gradient value of the pixel corresponding to the grid unit, store each average gradient value into the average gradient grid matrix, and generate an image grid matrix;

[0024] The image is converted into a normalized floating-point image. Based on a preset number of pyramid layers, a downsampling method is used to construct a first pyramid. Based on the first pyramid, an upsampling method is used to construct a second pyramid.

[0025] The imaging image, the image grid matrix, and the second pyramid are stored in an image data packet.

[0026] Furthermore, the streaming fusion method performs fusion processing on the fusion data set to obtain a fusion result image, including:

[0027] For N consecutively received imaging images, based on a preset number of fusion data blocks n, each time fusion is performed, image data packets of n imaging images are selected from the fusion data set for image fusion to obtain the fused image data packets for each fusion.

[0028] The fused image data packets generated during each fusion are stored in the fused data set, and the corresponding fused image data packets are used as a fused data block for secondary fusion in the next fusion.

[0029] Through multiple iterative fusion processes, until all image data packets of N imaging images have been fused at least once, the fused result image is output.

[0030] The number of fused data blocks refers to the number of image data packets when performing one image fusion.

[0031] Furthermore, the step of selecting n image data packets from the fused data set for image fusion to obtain the fused image data packet for each fusion includes:

[0032] For n imaging images, traverse the image grid matrix of the n image data packets;

[0033] By comparing the average gradient values ​​of the grid cells at corresponding positions of the n image grid matrices, the grid cell with the largest average gradient value is selected as the image fusion region;

[0034] Based on the image fusion region, a corresponding mask image is constructed for each of the n image grid matrices;

[0035] Traverse the second pyramid of n image data packets, and in the image fusion region of the mask image, perform pixel value fusion processing on the pixel values ​​of the n imaging images to obtain a pyramid fused image subset;

[0036] Upsampling is performed on the subset of the pyramid fused image to obtain the fused image;

[0037] Based on the fused image, the image information is extracted through the pre-preparation submodule to obtain the fused image data packet.

[0038] Furthermore, the external interaction unit includes an interface module, an interaction logic module, and a storage module;

[0039] The interface module is used to provide an interactive interface for users to perform interactive actions. The interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode. The imaging fusion mode includes manual fusion mode, automatic fusion mode and custom fusion mode.

[0040] The interaction logic module is used to convert the interaction action into an interaction command, and retrieve image information from the storage module to the interface module for software display;

[0041] The storage module is used to store the fusion configuration parameters corresponding to the interaction command, receive image information and imaging focus position, and the image information includes the imaging image and the fusion result image.

[0042] Furthermore, the image source unit includes a zoom module and an imaging module;

[0043] The zoom module is used to adjust the imaging focus position based on the imaging fusion mode. The imaging focus position is the relative physical position of the object plane and the imaging plane. Adjusting the imaging focus position includes adjusting the position of the object plane or the imaging plane, as well as adjusting the position of the object plane and the imaging plane.

[0044] The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

[0045] A multi-mode imaging fusion device is applied to the aforementioned multi-mode imaging fusion system.

[0046] A multi-mode imaging fusion method is applied to the aforementioned multi-mode imaging fusion system, the system comprising an image source unit, a fusion processing unit, an external interaction unit, and an image output unit, the method specifically comprising:

[0047] The imaging fusion mode is obtained through the external interaction unit and then fed back to the fusion processing unit.

[0048] The fusion processing unit enables the image source unit to acquire the imaging fusion mode, adjust the imaging focus position based on the imaging fusion mode, generate multiple imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit.

[0049] The fusion processing unit feeds back the imaging fusion mode to the image source unit, extracts image information from multiple imaging images to generate a fusion data set, and uses a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, which is then fed back to the image output unit.

[0050] The image output unit receives the fused result image and visualizes the fused result image.

[0051] Furthermore, the process of extracting image information from multiple imaging images is an asynchronous processing process.

[0052] The beneficial effects of the embodiments in this application compared with the prior art are:

[0053] This application discloses a multi-mode imaging fusion system. Before image fusion processing, image information of each imaging image is extracted, i.e., a pre-processing operation is performed to ensure that the effective information of each image can be fully utilized in the subsequent fusion process, thereby improving the accuracy and quality of the fusion result. Then, during the fusion process, a streaming fusion method is used to fuse the data of each extracted image data packet in real time, so that the imaging image can be processed in real time without waiting for all data to be collected before fusion, thereby improving processing efficiency. In addition, the fusion process of this application can adapt to the fusion implementation method in various scenarios. That is, the image fusion method in different imaging fusion modes is performed in the fusion processing module through pre-processing operations and streaming fusion method. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the optical principle of the depth-of-field effect provided in an embodiment of the present invention;

[0056] Figure 2This is a schematic diagram of the structure of a multi-mode imaging fusion system provided in an embodiment of the present invention;

[0057] Figure 3 This is a flowchart illustrating the preparatory operations provided in an embodiment of the present invention;

[0058] Figure 4 This is a flowchart illustrating the fusion process of two image data packets provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the process of imaging and image fusion under different imaging fusion modes provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of a feasible embodiment of the adjustable back-end optical component provided by the present invention;

[0061] Figure 7 This is a flowchart illustrating a multi-mode imaging fusion method provided in an embodiment of the present invention. Detailed Implementation

[0062] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0063] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0064] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0065] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0066] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0068] Please see Figures 1-6 As shown, the present invention is a multi-mode imaging fusion system, comprising: an image source unit, a fusion processing unit, an external interaction unit, and an image output unit;

[0069] The external interaction unit is used to acquire the imaging fusion mode and feed the imaging fusion mode back to the fusion processing unit;

[0070] The image source unit is used to obtain the imaging fusion mode through the fusion processing unit, adjust the imaging focus position based on the imaging fusion mode, generate multiple imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit.

[0071] The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, extract image information from multiple imaging images to generate a fusion data set, perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feed back the fusion result image to the image output unit.

[0072] The image output unit is used to receive the fused result image and visualize the fused result image.

[0073] The multi-mode imaging fusion system of this application is based on the optical principle of depth-of-field effect, specifically, as follows: Figure 1As shown, there are points B1 and B2 on object planes A1 and A2, respectively. Points B1 and B2 are imaged as B1' and B2' on imaging planes A1' and A2', respectively. However, when imaging is performed by imaging devices such as camera sensors, it is usually based on a reference image plane A'. On the reference image plane A', the images corresponding to B1' and B2' are shown as the blur spots Z'. In a feasible embodiment, when the size of the blur spots on the reference image plane A' is not perceptible to the human eye, the objects on object planes A1 and A2 are clearly visible on the reference image plane A'. It can be understood that when the size of the light spot imaged on the reference image plane A' is not perceptible to the human eye within the depth of field, the objects within the depth of field are clearly visible on the imaging device. Therefore, in this application, by controlling the relative motion of object plane A and reference image plane A', an image set with a depth of field range exceeding the original optical path is obtained, and then a clear image with a larger depth of field range is synthesized by performing a fusion operation on the image set.

[0074] In this application, the imaging focus position is adjusted according to the imaging fusion mode. It is understood that there are multiple imaging fusion modes, and the method of adjusting the imaging focus position differs depending on the imaging fusion mode, which can adapt to various user scenarios. The imaging focus position is the relative physical position of the object plane and the imaging plane, i.e., the relative position of the controlled object plane A and the reference image plane A'. Multiple imaging images generated during the imaging focus position adjustment process are acquired, and image information from these images is extracted to generate a fusion data set. A streaming fusion method is used to fuse the fusion data set to obtain the fused result image. It is understood that in this application, during the... Before image fusion processing, image information of each imaging image is extracted to ensure that the effective information of each image can be fully utilized in the subsequent fusion process, thereby improving the accuracy and quality of the fusion result. Then, during the fusion process, a streaming fusion method is adopted to fuse the data of each extracted image data packet in real time, so that the imaging image can be processed in real time without waiting for all data to be collected before fusion, thereby improving processing efficiency. In addition, the fusion process of this application can adapt to the fusion implementation method in various scenarios. That is, the image fusion method in different imaging fusion modes is performed in the fusion processing module through pre-preparation operations and streaming fusion method.

[0075] In some embodiments, the fusion processing unit includes a fusion logic module, an image cache module, and a fusion processing module;

[0076] The fusion logic module is used to receive and feed back the interaction instructions from the external interaction unit to the image cache module and the image source unit;

[0077] The image caching module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module;

[0078] The fusion processing module is used to receive the fusion data set, and to perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and to feed back the fusion result image to the image output unit and the external interaction unit.

[0079] When the above-mentioned multi-mode imaging fusion system is applied to a multi-mode imaging fusion device, the fusion processing unit is a Linux program running on a DSP chip, which is not limited by the physical device it supports or the operating system it runs.

[0080] Specifically, such as Figure 2 As shown, the fusion logic module receives interaction commands and corresponding fusion configuration parameters from the external interaction unit, and simultaneously feeds back the relevant interaction commands to the image source unit, enabling the image source unit to complete the imaging of the source image according to the interaction commands and generate an imaging image. The generated imaging image and the source image are output to the image cache module for image information extraction and image data packet caching. The fusion data set in the image cache module is output to the fusion processing module for fusion processing, and finally, a fusion result image is generated. The generated fusion result image is fed back to the external interaction unit and the image output unit, thereby realizing the visualization of the fusion result image.

[0081] In this embodiment, the feedback modes of the fusion logic module include passive response feedback and active upload feedback. The two feedback modes work together to realize information interaction between the fusion processing unit, the image source unit, and the external interaction unit.

[0082] In some embodiments, the image caching module includes a data receiving submodule, a pre-processing submodule, and a data collection submodule;

[0083] The data receiving submodule is used to receive source images and imaging images;

[0084] The pre-preparation submodule is used to extract image information from multiple imaging images to generate image data packets;

[0085] The data set submodule is used to generate a fused data set based on multiple image data packets.

[0086] In this embodiment, the image caching module includes pre-processing operations for each imaging image before fusion, namely, extracting image information from multiple imaging images and finally encapsulating them into image data packets. It can be understood that the image data packets are other image data and parameter data generated by image processing of the imaging images. By separating the image processing operations before imaging image fusion processing to the image caching unit, asynchronous operations between image information extraction of imaging images and fusion of each imaging image are realized. Thus, the image fusion processing process will not block the continuous reception of imaging images by the image caching module, realizing real-time output of fused data. Therefore, the image caching module includes a data receiving submodule, a pre-processing submodule, and a data collection submodule. Through the cooperation of the above three submodules, the pre-processing operations for imaging images and the caching of corresponding image information are realized.

[0087] In this embodiment, the number of image data packets in the input fusion data set can be a fixed value of 2 or any value greater than 2. When the number of image data packets in the input fusion data set is any value greater than 2, the image data packets generated after image information extraction from each imaging image can be stored in the fusion data set. During the fusion process, the relevant image data packets are called from the fusion data set for streaming fusion, thereby realizing the fusion of image data from multiple imaging images.

[0088] In this embodiment, when the image storage module obtains the imaging image fed back by the image source module, it synchronously performs pre-fusion preparation operations on the imaging image. The pre-fusion preparation operations of different imaging images in the image storage module are asynchronous operations. When a new imaging image is input before the previous pre-fusion operation is completed, it can be processed by creating a new thread without waiting for the previous pre-fusion operation to complete, thereby improving processing efficiency.

[0089] In some embodiments, the image data packet includes an image, an image grid matrix, and a second pyramid;

[0090] The step of extracting image information from multiple imaging images to generate image data packets includes:

[0091] An average gradient grid matrix is ​​constructed based on the preset grid specifications of the grid cells;

[0092] If the image being imaged is a color image, then the image being imaged is converted to a grayscale image;

[0093] The grayscale image or the imaging image is preprocessed, and a gradient operator is used to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaging image.

[0094] Calculate the average gradient value of the pixel corresponding to the grid unit, store each average gradient value into the average gradient grid matrix, and generate an image grid matrix;

[0095] The image is converted into a normalized floating-point image. Based on a preset number of pyramid layers, a downsampling method is used to construct a first pyramid. Based on the first pyramid, an upsampling method is used to construct a second pyramid.

[0096] The imaging image, the image grid matrix, and the second pyramid are stored in an image data packet.

[0097] In this embodiment, when extracting image information from the imaging image, if the imaging image is a color image, it is necessary to first convert the imaging image into a grayscale image and then construct the image grid matrix, thereby reducing the data dimension and significantly reducing the computational complexity and time cost of subsequent image information extraction.

[0098] In this embodiment, the average gradient value of the imaging image is combined with a grid matrix to construct an average gradient grid matrix that describes the gradient change relationship between pixels in the imaging image. In addition, the imaging image is downsampled and upsampled to construct the average gradient grid matrix for detail extraction and image reconstruction. The average gradient grid matrix and the second pyramid are the data basis for fusing multiple imaging images.

[0099] Specifically, for the average gradient grid matrix, before construction, the grid size of the grid unit needs to be preset according to the resolution and feature scale of the imaging image. In a preferred embodiment, the size of a single grid unit is 100×100 pixels, that is, the image is divided into multiple grid units of size 100×100 pixels. For an imaging image with a size of 1920×1080, the size of the final generated average gradient grid matrix is ​​20×11, that is, obtained by rounding up 1920 / 100 and 1080 / 100. In addition, for the second pyramid, before construction, the number of pyramid layers needs to be preset according to the size requirements of the target layer image. In a preferred embodiment, the number of pyramid layers is 3.

[0100] Specifically, such as Figure 3As shown, when generating an image grid matrix using the average gradient grid matrix, the grayscale image or imaging image is preprocessed, such as by performing Gaussian filtering to reduce noise and interference. Then, the gradient value of each pixel in the preprocessed grayscale image or imaging image is calculated. Specifically, the gradient value of each pixel can be obtained by using gradient operators, such as the Sobel operator or the Prewitt operator. For each grid cell, its corresponding average gradient value is calculated, and the corresponding average gradient value is filled into the grid matrix, so that each element of the average gradient grid matrix represents the gradient change value between adjacent pixels, thereby generating the image grid matrix.

[0101] In this embodiment, as Figure 3 As shown, when constructing the second pyramid, the imaging image is converted into a normalized floating-point image. Specifically, the pixel values ​​of the imaging image are converted into floating-point numbers between 0 and 1. Specifically, the maximum and minimum values ​​of the pixel values ​​in the imaging image are calculated, and each pixel value is converted into a floating-point number between 0 and 1 based on the conversion formula. In a feasible embodiment, the conversion formula is: (pixel value - minimum value) / (maximum value - minimum value).

[0102] In this embodiment, the first pyramid constructed using the downsampling method is a Gaussian image pyramid. The Gaussian pyramid is used for image downsampling and scale space representation. Specifically, the size representation of each pyramid layer is obtained by filtering the previous layer image using a Gaussian blur kernel, thereby generating the first pyramid. Then, based on the first pyramid, the second pyramid is constructed using the upsampling method. The second pyramid is a Laplacian image pyramid, constructed by the difference between the image of each layer in the Gaussian image pyramid and the image of the previous layer. It is mainly used for detail extraction and reconstruction of the imaging image.

[0103] Understandably, in this embodiment, the construction of the image grid matrix and the second pyramid can be carried out simultaneously, thereby making full use of computing resources and reducing the overall processing time. The images are then stored in the image data packet, which also contains the unprocessed imaging image, facilitating quick access to the relevant data during the subsequent fusion process.

[0104] In some embodiments, the streaming fusion method performs fusion processing on the fusion data set to obtain a fusion result image, including:

[0105] For N consecutively received imaging images, based on a preset number of fusion data blocks n, each time fusion is performed, image data packets of n imaging images are selected from the fusion data set for image fusion to obtain the fused image data packets for each fusion.

[0106] The fused image data packets generated during each fusion are stored in the fused data set, and the corresponding fused image data packets are used as a fused data block for secondary fusion in the next fusion.

[0107] Through multiple iterative fusion processes, until all image data packets of N imaging images have been fused at least once, the fused result image is output.

[0108] The number of fused data blocks refers to the number of image data packets when performing one image fusion.

[0109] In this embodiment, N consecutively received imaging images are labeled as ImageInput1, ImageInput2, ..., ImageInputN. Then, preparatory actions are performed on each image to generate image data packets data1, data2, ..., dataN. The fused image data packets generated each time are labeled as g_data1, g_data2, ..., and the corresponding fused image data packets are stored in the fused data set for use as a fused data block in the next fusion.

[0110] Since the number of image data packets in the input fusion dataset can be a fixed value of 2 or any value greater than 2, when the number of image data blocks in the fusion dataset is a fixed value of 2, one of the data packets in the fusion dataset is an empty fusion image data packet g_data1. During the fusion process, this fusion image data packet g_data1 is iteratively assigned the value g_data2 or g_data3, etc. Based on this, when the number of input fusion datasets is fixed at 2, the problem of excessive time consumption caused by multiple loops of fusion image data packet pre-preparation operations can be avoided. When the number of input fusion datasets is greater than 2, since the pre-preparation operations for each imaging image have been completed through the image buffer unit and cached in the fusion dataset, further fusion processing can be performed on the image data blocks to further improve the processing speed.

[0111] In this embodiment, before data fusion, the number of fusion data blocks is determined. A reasonable setting of the number of fusion data blocks can effectively improve the real-time performance of fusion of various imaging images. In some preferred embodiments, the number of fusion data blocks is 2.

[0112] In this embodiment, we will take a case where the number of fused data blocks is 2 and the number of image data packets in the input fused data set is a fixed value of 2 as an example. Specifically, the data1 of the received ImageInput1 in the fused data set is assigned to the empty fused data unit g_data1. The data2 of the next imaging image ImageInput2 is then input into the fused data set for fusion. It can be understood that a single imaging image cannot be used for fusion. Therefore, the data1 of the first imaging image ImageInput1 is used as the fused image data packet g_data1 in the fused data set, and the image fusion is performed on the data2 of the next imaging image ImageInput2 to generate a new fused image data packet g_data2. That is, g_data2 = g_data1 + data2, where g_data2 is a fusion data block for the next fusion. When data3 of the third imaging image ImageInput3 is input into the fusion data set, g_data2 and data3 are fused to generate a fusion image data packet g_data3. At this time, g_data3 = g_data2 + data3. In this streaming fusion method, each image data packet in the input fusion data set is iteratively fused until all imaging images have completed the corresponding fusion operation. At this time, the fusion image corresponding to g_dataN in the last fusion operation g_dataN = g_data(N-1) + dataN is fed back as the fusion result image to the image output unit and the external interaction unit.

[0113] Furthermore, for embodiments where the number of fused data blocks is greater than 2, and the number of image data packets in the input fused data set is a fixed value of 2, a feasible implementation is as follows: Image data blocks that do not meet the fusion requirements are stored in the fused image data packet g_data1 for fusion preparation. When the last fused data block is input into the fused data set, the data fusion operation is performed. Specifically, if the number of fused data blocks is 3, the data1 and data2 of the first and second imaging images are stored in the fused image data packet g_data1. When the data3 of the third imaging image is input into the fused data set, data1, data2, and data3 are fused to generate the fused image data packet g_data2, and the data is iteratively assigned to g_data1, waiting for the input of the image data packet of the next imaging image.

[0114] In some embodiments, the step of selecting n image data packets from the fusion data set for image fusion to obtain the fused image data packet for each fusion includes:

[0115] For n imaging images, traverse the image grid matrix of the n image data packets;

[0116] By comparing the average gradient values ​​of the grid cells at corresponding positions of the n image grid matrices, the grid cell with the largest average gradient value is selected as the image fusion region;

[0117] Based on the image fusion region, a corresponding mask image is constructed for each of the n image grid matrices;

[0118] Traverse the second pyramid of n image data packets, and in the image fusion region of the mask image, perform pixel value fusion processing on the pixel values ​​of the n imaging images to obtain a pyramid fused image subset;

[0119] Upsampling is performed on the subset of the pyramid fused image to obtain the fused image;

[0120] Based on the fused image, the image information is extracted through the pre-preparation submodule to obtain the fused image data packet.

[0121] In this embodiment, taking a fusion data block count of 2 as an example, the image fusion process is described in detail. Specifically, the image grid matrices in the two image data packets are traversed, and the average gradient values ​​of corresponding grid cells in the two image grid matrices are compared. The grid cell with the largest average gradient value is selected as the image fusion region. It can be understood that the grid cell with the largest average gradient value usually contains more image details and features, so it is selected as a potential fusion region for subsequent image fusion. Further, for the two image grid matrices, a corresponding mask image is constructed according to the image fusion region corresponding to the corresponding image grid matrix, which is used for quantization marking of the image fusion region. Based on the mask image, the second pyramid in the two image data packets is traversed, and the pixel values ​​of the corresponding image fusion region in the two imaging images are fused to obtain a pyramid fusion image subset. The pyramid fusion image subset is then upsampled to obtain the fused image generated by the fusion of the two imaging images. For fusions that are not the last iteration, the image information of the fused image is extracted by the pre-preparation submodule to generate a fused image data packet, which is stored in the fusion data set and fused with the next image data packet for secondary fusion. For the last iteration, the fused image is the fusion result image.

[0122] In some embodiments, the method for performing pixel value fusion on a portion of the corresponding image fusion region in the imaging image includes, but is not limited to, using a weighted fusion method.

[0123] Specifically, such as Figure 4As shown, two image data packets are labeled A and B. The average gradient values ​​of the grid cells at corresponding positions in the two image grid matrices are compared, and the grid cell with the largest average gradient value is selected as the image fusion region. Multiple image fusion regions belonging to the same image grid matrix are then integrated into a single mask image. Specifically, this involves: constructing corresponding index image matrices for image data packets A and B of the two imaging images, where the index image matrix is ​​adapted to the image grid matrix of the imaging image, and configuring a first pixel value (specifically 0) for each grid cell in the index image matrix; comparing the average gradient values ​​of the corresponding grid cells in the image grid matrices of the two image data packets, and selecting the grid cell with the largest average gradient value as the image fusion region. The first pixel value of the grid cell corresponding to the image fusion region in the indexed image matrix is ​​modified to the second pixel value, specifically 1. At the same time, a mask image is constructed based on the indexed image matrix. Specifically, for the grid cells with a pixel value of 1 in the indexed image matrix, all pixel values ​​of the corresponding grid cells in the mask image are set to 1. Then, the second pyramids of image data packets A and B are traversed respectively. The parts with pixel values ​​of 1 in the mask image are pixel fused at the corresponding pyramid level. Preferably, the pixel fusion is performed by a pixel value weighted average method to obtain a fused pyramid fused image subset. Then, based on the fused pyramid fused image subset, upsampling is performed to obtain the fused image.

[0124] In some embodiments, the external interaction unit includes an interface module, an interaction logic module, and a storage module;

[0125] The interface module is used to provide an interactive interface for users to perform interactive actions. The interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode. The imaging fusion mode includes manual fusion mode, automatic fusion mode and custom fusion mode.

[0126] The interaction logic module is used to convert the interaction action into an interaction command, and retrieve image information from the storage module to the interface module for software display;

[0127] The storage module is used to store the fusion configuration parameters corresponding to the interaction command, receive image information and imaging focus position, and the image information includes the imaging image and the fusion result image.

[0128] When the above-mentioned multi-mode imaging fusion system is applied to a multi-mode imaging fusion device, the external interaction unit is a Linux program running on a DSP chip. It is not limited by the physical device it supports or the operating system it runs. It can be an Android program on a mobile device, an iOS program on an iOS device, an exe program on a Windows system device, or a supported browser web program. It communicates with the fusion processing unit, and the communication method is not limited to a specific protocol. Specifically, the communication content between the external interaction unit and the fusion processing unit includes, but is not limited to, fusion mode switching information, fusion area configuration information, fusion process control information, fusion intermediate image feedback information, and process operation feedback information.

[0129] In this embodiment, the user can select the fusion processing mode and configure the fusion configuration parameters in the corresponding fusion processing mode through the interface unit, such as the zoom range, zoom amount, zoom speed, fusion weight, specified fusion area location information, process node jump configuration, etc. The storage unit is responsible for storing the fusion configuration parameters corresponding to the interaction command, receiving the imaging image, receiving the fusion result screen, receiving the focus position information, etc. The interaction logic unit is responsible for converting the user's interaction actions into interaction commands, starting the corresponding running process, and retrieving the imaging image and fusion result image from the storage unit to the interface unit for software display.

[0130] In this embodiment, the mode of controlling the relative motion between the object plane A and the reference image plane A', i.e. the imaging fusion mode, has three modes: manual fusion mode, automatic fusion mode, and custom fusion mode. Among them, custom focus range fusion is a set of implementations under different focus ranges, such as 1 / 3 focus range, 2 / 3 focus range, or focus ranges composed of different focus start and end positions. Based on the above three imaging fusion modes, the fusion process of this application can adapt to different fusion implementation mechanisms in various scenarios. That is, the above three imaging fusion modes can all use the pre-preparation operation of the imaging image and the streaming fusion method in the above fusion processing unit to perform image fusion.

[0131] Specifically, such as Figure 5As shown, in manual fusion mode, adjustments can be made through interface controls on the interactive interface provided by the external interaction unit, such as the position slider, to directly adjust the imaging focus position in the image source unit, thereby outputting the corresponding imaging image for real-time fusion output. In other words, in manual fusion mode, corresponding interactive commands must be provided through the external interaction unit. Specifically, after adjusting the imaging focus position in manual fusion mode, the fusion logic unit of the fusion processing unit initiates a stop timing action, and fusion stops after the timing expires, thereby reducing system load. If a new zoom adjustment occurs during the timing process, the timing action is canceled. In automatic fusion mode, image sharpness detection is provided. When the upward trend in sharpness is detected to disappear continuously, the fusion process automatically stops. In one embodiment, the sum of the Sobel gradient values ​​in the horizontal and vertical directions is used as the sharpness value. When the upward trend in sharpness is detected to disappear continuously five times, the fusion process automatically stops. In the custom focus range fusion mode, the fusion logic unit of the fusion processing unit is responsible for controlling the zoom module according to the preset range or step value to achieve focus fusion within the focus range. The fusion action is stopped only when the focus within the focus range is completed.

[0132] In some embodiments, the image source unit includes a zoom module and an imaging module;

[0133] The zoom module is used to adjust the imaging focus position based on the imaging fusion mode. The imaging focus position is the relative physical position of the object plane and the imaging plane. Adjusting the imaging focus position includes adjusting the position of the object plane or the imaging plane, as well as adjusting the position of the object plane and the imaging plane.

[0134] The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

[0135] In this embodiment, the zoom module adjusts the imaging focus position, and the imaging focus position corresponds to... Figure 1 The relative motion between the control object plane A and the reference image plane A' shown refers specifically to the relative physical position of the object plane and the imaging plane. In addition, the source image is converted into an imaging image by the imaging module and output to the fusion processing unit for image fusion processing. It can be understood that the output format of the imaging image is a digital image signal, which is beneficial for image processing.

[0136] Specifically, the zoom unit is an external optical path device that acts on the imaging optical path, and the imaging unit is an optoelectronic device used to acquire the final image of the optical path. When the above-mentioned multi-mode imaging fusion system is applied to the multi-mode imaging fusion device, the zoom unit can be a motor focusing device, and the imaging unit can be a global shutter sensor, a linear sensor, or a rolling shutter camera, etc., without being limited by the specific physical device form or type.

[0137] In some embodiments, adjusting the imaging focus position can specifically involve adjusting, for example... Figure 1 The object plane or imaging plane shown is described above. The external optical path device includes a front-end optical component, a rear-end optical component, and a driving component. The front-end optical component is typically located at the very front of the optical path imaging device and is responsible for receiving the light rays from the object, i.e., the light signal. The front-end optical component includes, but is not limited to, a lens and a compensation lens. In one feasible embodiment, the compensation lens is connected to the driving component. By setting a step value and a step number for the driving component, the compensation lens moves during focusing, thereby adjusting the front-end optical component and achieving a change in the imaging focus position. In another feasible embodiment, the movement of the compensation lens is independent of… The driving component, which is a connection structure associated with the supplementary lens through manual control, causes the supplementary lens to move, thereby achieving focusing. Specifically, it can be adapted to the imaging fusion mode of manual fusion mode. When in manual fusion mode, the external interaction unit sets a trigger timer and sends fusion control commands to the fusion logic module of the fusion processing unit at specified time intervals. The fusion logic module then sends a command containing only trigger information to the zoom module of the image source unit. At this time, the zoom module only plays the role of triggering the image output of the imaging module. The output of the source image is ultimately controlled by the external interaction unit, and the output process is controlled by the timer of the external interaction unit.

[0138] For the back-end optical components, the purpose is to convert the light signals captured by the front-end optical components into electrical signals to form a digital image. The back-end optical components include, but are not limited to, image sensors and optical interfaces. The optical interface is used to connect the back-end and front-end optical components. In specific applications, the optical interface can be a USB-C interface or other interfaces, used to fix the back-end optical components to the lens in an optical imaging device or to embed the back-end optical components as sub-modules in an eyepiece camera, etc. In another feasible embodiment, the image sensor is driven by a driving component, thereby adjusting the plane on which the image sensor is located, achieving movement of the imaging plane. Specifically, as shown... Figure 6As shown, the reference focus point P1' corresponding to object point P1 is located between imaging planes A1' and A2'. When the driving component moves, the image sensor plane undergoes the same displacement under the action of the driving component, thereby obtaining imaging data passing through plane P1'. During the fusion process, through image gradient detection, the detailed information of P1' can be retained in the image. Similarly, for any point located within the object planes A1 to A2, the above processing can also be performed to form a clear fused image of the imaging range A1' to A2' of planes A1 to A2. In another feasible embodiment, the movement of the image sensor does not depend on the driving component, that is, it is achieved by manually controlling the connection structure associated with the image sensor. The specific implementation process is similar to the way the connection structure associated with the supplementary lens in the front-end optical component is manually controlled for focus adjustment, and will not be described in detail here.

[0139] The driving component can be a motor or other device capable of moving the supplementary lens or image sensor. Its specific structure will not be described in detail here. By using both driving components to control the movement of the supplementary lens or image sensor and manual control of its movement, the multi-mode imaging fusion system of this application is compatible with optical imaging devices of varying costs, providing a low-cost expansion method. The lens used in the optical imaging device can be a non-fixed-focus lens, a zoom lens, or an autofocus lens. The optical imaging device can also be a microscope, telescope, etc., with autofocus functionality. Therefore, this multi-mode fusion imaging system has good scalability and adaptability. It is understood that the optical imaging device is a multi-mode imaging fusion device that applies this multi-mode imaging fusion system.

[0140] Specifically, when adjusting the imaging focus position, it can be achieved by adjusting the front-end optical components, adjusting the rear-end optical components, or adjusting both the front-end and rear-end optical components. It can be understood that the final manifestation of the zoom module's adjustment of the imaging focus position is that the plane of the subject and the imaging plane are physically displaced. The adjustment of the plane of the subject is achieved by adjusting the front-end optical components, and the adjustment of the imaging plane is achieved by adjusting the rear-end optical components.

[0141] In this invention, the zoom unit is not limited to the actual position of the optical path of the optical component, nor is it limited to the specific method of achieving zoom adjustment by combining multiple optical components. The aforementioned front-end optical component and rear-end optical component are described as one embodiment of zoom adjustment, and not as a limitation thereof.

[0142] The image output unit specifically includes a screen display module, which is responsible for directly displaying the image and presenting the fused image in the form of a floating window, an embedded window, a switching window, or multiple windows.

[0143] This application also provides a multi-mode imaging fusion device, including the above-mentioned multi-mode imaging fusion system.

[0144] Specifically, the aforementioned multi-mode imaging fusion device includes a memory and a processor, as well as a computer program stored in the memory. When the computer program is executed on the processor, the aforementioned multi-mode imaging fusion method can be implemented. The multi-mode imaging fusion device can be an optical imaging device with focusing imaging function, such as a microscope or a telescope. By applying the aforementioned multi-mode imaging fusion system to the multi-mode imaging fusion device, automatic focusing imaging of the microscope or telescope can be achieved.

[0145] Please see Figure 7 As shown, this application also provides a multi-mode imaging fusion method applied to the aforementioned multi-mode imaging fusion system. The system includes an image source unit, a fusion processing unit, an external interaction unit, and an image output unit. The method specifically includes:

[0146] S100: Obtain the imaging fusion mode through the external interaction unit, and feed the imaging fusion mode back to the fusion processing unit;

[0147] S200. Through the fusion processing unit, the image source unit acquires the imaging fusion mode, adjusts the imaging focus position based on the imaging fusion mode, generates multiple imaging images based on the imaging focus position, and feeds the imaging images back to the fusion processing unit.

[0148] S300: The imaging fusion mode is fed back to the image source unit through the fusion processing unit, and image information is extracted from multiple imaging images to generate a fusion data set. The fusion data set is fused using a streaming fusion method to obtain a fusion result image, and the fusion result image is fed back to the image output unit.

[0149] S400: Receive the fused result image through the image output unit and visualize the fused result image.

[0150] In this application, the imaging fusion mode and corresponding fusion configuration parameters are obtained through the interface module of the external interaction unit. The interaction logic module converts the imaging fusion mode into an interaction command and feeds it back to the fusion logic module of the fusion processing unit. The fusion logic module feeds the interaction command back to the zoom module of the image source unit. The zoom module adjusts the imaging focus position according to the imaging fusion mode. Then, the imaging module of the image source unit converts the source image into an imaging image based on the imaging focus position and feeds the imaging image back to the image cache module of the fusion processing unit. The image cache module extracts image information from the imaging image and finally generates a fusion data set. The image cache module outputs the fusion data set to the fusion processing module for image fusion processing to generate a fusion result image. The final fusion result image is output to the image output unit for visualization. In addition, the fusion process of each imaging image is an independent process. The imaging image is always cached during the fusion process. Specifically, the fusion result image and the imaging image are also output to the storage module of the external interaction unit for storage, which is used to retrieve them to the interface unit for software display.

[0151] In one embodiment, the process of extracting image information from multiple imaging images is an asynchronous processing process.

[0152] In this embodiment, the extraction of image information from the imaging image and the fusion of different imaging images are asynchronous operations, which prevents the image fusion process from blocking the continuous reception of imaging images by the image caching module. In addition, the pre-preparation operations of different imaging images in the image storage module are also asynchronous operations. When a new imaging image is input before the previous pre-preparation operation is completed, it can be processed by creating a new thread without waiting for the previous pre-preparation operation to complete, thereby improving processing efficiency.

[0153] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-mode imaging fusion system, characterized in that, include: Image source unit, fusion processing unit, external interaction unit, and image output unit; The external interaction unit is used to acquire the imaging fusion mode and feed the imaging fusion mode back to the fusion processing unit; The image source unit is used to obtain the imaging fusion mode through the fusion processing unit, adjust the imaging focus position based on the imaging fusion mode, generate multiple imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit. The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, extract image information from multiple imaging images to generate a fusion data set, perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feed back the fusion result image to the image output unit. The image output unit is used to receive the fused result image and visualize the fused result image; The method employs a streaming fusion approach to process the fusion data set and obtain a fused image. This includes: for N consecutively received imaging images, based on a preset number of fusion data blocks n, selecting image data packets from the fusion data set of n imaging images for image fusion at each fusion stage, obtaining a fused image data packet for each fusion. The image data packet includes the imaging image, an image grid matrix, and a second pyramid. The fused image data packet generated at each fusion stage is stored in the fusion data set, and the corresponding fused image data packet is used as a fusion data block for secondary fusion at the next fusion stage. Through multiple iterative fusions, until all image data packets of the N imaging images have undergone at least one fusion, the fused image is output. The number of fusion data blocks is the number of image data packets used in one image fusion. The step of selecting image data packets of n imaging images from the fusion data set for image fusion to obtain fused image data packets for each fusion includes: for the n imaging images, traversing the image grid matrix of the n image data packets; comparing the average gradient values ​​of the grid cells at corresponding positions in the n image grid matrices, and selecting the grid cell with the largest average gradient value as the image fusion region; constructing a corresponding mask image for the n image grid matrices based on the image fusion region; traversing the second pyramid of the n image data packets, and performing pixel value fusion processing on the pixel values ​​of the n imaging images in the image fusion region of the mask image to obtain a pyramid fused image subset; performing upsampling on the pyramid fused image subset to obtain a fused image; and obtaining a fused image data packet by extracting image information through a pre-preparation submodule based on the fused image.

2. The system as described in claim 1, characterized in that, The fusion processing unit includes a fusion logic module, an image cache module, and a fusion processing module; The fusion logic module is used to receive and feed back the interaction instructions from the external interaction unit to the image cache module and the image source unit; The image caching module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module; The fusion processing module is used to receive the fusion data set, and to perform fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and to feed back the fusion result image to the image output unit and the external interaction unit.

3. The system as described in claim 2, characterized in that, The image caching module includes a data receiving submodule, a pre-processing submodule, and a data collection submodule; The data receiving submodule is used to receive source images and imaging images; The pre-preparation submodule is used to extract image information from multiple imaging images to generate image data packets. The image data packets include imaging images, image grid matrices, and a second pyramid. The data set submodule is used to generate a fused data set based on multiple image data packets.

4. The system as described in claim 3, characterized in that, The step of extracting image information from multiple imaging images to generate image data packets includes: An average gradient grid matrix is ​​constructed based on the preset grid specifications of the grid cells; If the image being imaged is a color image, then the image being imaged is converted to a grayscale image; The grayscale image or the imaging image is preprocessed, and a gradient operator is used to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaging image. Calculate the average gradient value of the pixel corresponding to the grid unit, store each average gradient value into the average gradient grid matrix, and generate an image grid matrix; The image is converted into a normalized floating-point image. Based on a preset number of pyramid layers, a downsampling method is used to construct a first pyramid. Based on the first pyramid, an upsampling method is used to construct a second pyramid. The imaging image, the image grid matrix, and the second pyramid are stored in an image data packet.

5. The system as described in claim 1, characterized in that, The external interaction unit includes an interface module, an interaction logic module, and a storage module; The interface module is used to provide an interactive interface for users to perform interactive actions. The interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode. The imaging fusion mode includes manual fusion mode, automatic fusion mode and custom fusion mode. The interaction logic module is used to convert the interaction action into an interaction command, and retrieve image information from the storage module to the interface module for software display; The storage module is used to store the fusion configuration parameters corresponding to the interaction command, receive image information and imaging focus position, and the image information includes the imaging image and the fusion result image.

6. The system as described in claim 1, characterized in that, The image source unit includes a zoom module and an imaging module; The zoom module is used to adjust the imaging focus position based on the imaging fusion mode. The imaging focus position is the relative physical position of the object plane and the imaging plane. Adjusting the imaging focus position includes adjusting the position of the object plane or the imaging plane, as well as adjusting the position of the object plane and the imaging plane. The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

7. A multi-mode imaging fusion device, characterized in that, Includes the multi-mode imaging fusion system as described in any one of claims 1-6.

8. A multi-mode imaging fusion method, characterized in that, The method is applied to the multi-mode imaging fusion system according to any one of claims 1-6, the system comprising an image source unit, a fusion processing unit, an external interaction unit, and an image output unit, and specifically includes: The imaging fusion mode is obtained through the external interaction unit and then fed back to the fusion processing unit. The fusion processing unit enables the image source unit to acquire the imaging fusion mode, adjust the imaging focus position based on the imaging fusion mode, generate multiple imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit. The fusion processing unit feeds back the imaging fusion mode to the image source unit, extracts image information from multiple imaging images to generate a fusion data set, and uses a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, which is then fed back to the image output unit. The image output unit receives the fused result image and visualizes the fused result image. The process of extracting image information from multiple imaging images is an asynchronous processing process.

Images