Multi-mode imaging fusion system, device and method

Abstract

The invention is suitable for the technical field of imaging processing, and provides a multi-mode imaging fusion system, device and method, and the system comprises an external interaction unit which is used for obtaining an imaging fusion mode; the image source unit is used for acquiring the imaging fusion mode through the fusion processing unit, adjusting an imaging focusing position based on the imaging fusion mode, generating a plurality of imaging images based on the imaging focusing position, and feeding back the imaging images to the fusion processing unit; the fusion processing unit is used for performing image information extraction on the plurality of imaging images to generate a fusion data set, and performing fusion processing on the fusion data set by adopting a streaming fusion method to obtain a fusion result image; and the image output unit is used for visually presenting the fusion result image. The multi-mode imaging fusion system has a real-time fusion method and a fusion implementation method suitable for various scenes.

Description

Technical Field

[0001] The present application belongs to the field of imaging processing technology, and in particular, relates to a multi-mode imaging fusion system, device and method. Background Art

[0002] Imaging processing systems are usually used in many fields such as optics, teaching, research and civil use. They are usually externally connected, embedded or split in various optical path system devices. The above-mentioned optical path system devices include but are not limited to embedded stereo microscope cameras, embedded biological microscope cameras, embedded spectroscopic microscope cameras, external cameras for the optical path at the eyepiece end of the microscope, split cameras for the optical path at the objective end of the microscope, and electronic eyepieces at the end of the telescope.

[0003] Most of the imaging processing systems used in the above-mentioned optical path system devices in the prior art only have basic camera property adjustment functions. In addition, for imaging devices with the function of adjusting the imaging focus position, most of them only realize the single control of the focus distance and cannot provide a composite image including multiple layers of focus position images, thereby limiting the diversity of imaging and the acquisition of depth information. Furthermore, in application scenarios that require real-time acquisition and processing of multiple layers of focus position images, the above-mentioned imaging processing system is unable to cope with the situation and cannot provide sufficient flexibility and real-time performance.

[0004] Based on this, the present application proposes a multi-mode imaging fusion system with a real-time fusion method and different fusion implementation mechanisms that are adaptable to a variety of scenarios. Summary of the invention

[0005] The embodiments of the present application provide a multi-mode imaging fusion system, device and method, which can solve one of the above-mentioned problems in the prior art.

[0006] A multi-mode imaging fusion system, comprising: an image source unit, a fusion processing unit, an external interaction unit and an image output unit;

[0007] The external interaction unit is used to obtain an imaging fusion mode and feed the imaging fusion mode back to the fusion processing unit;

[0008] The image source unit is used to obtain the imaging fusion mode through the fusion processing unit, adjust the imaging focus position based on the imaging fusion mode, generate a plurality of imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit;

[0009] The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, and extract image information from the plurality of imaging images to generate a fusion data set, adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed back the fusion result image to the image output unit;

[0010] The image output unit is used to receive the fusion result image and visualize the fusion result image.

[0011] Furthermore, the fusion processing unit includes a fusion logic module, an image buffer module and a fusion processing module;

[0012] The fusion logic module is used to receive and feed back the interaction instruction of the external interaction unit to the image cache module and the image source unit;

[0013] The image cache module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module;

[0014] The fusion processing module is used to receive the fusion data set, and adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed the fusion result image back to the image output unit and the external interaction unit.

[0015] Further, the image cache module includes a data receiving submodule, a pre-preparation submodule and a data collection submodule, and the image data packet includes an imaging image, an image grid matrix and a second pyramid;

[0016] The data receiving submodule is used for receiving the source image and the imaging image;

[0017] The pre-preparation submodule is used to extract image information from the plurality of imaging images respectively to generate image data packets;

[0018] The data set submodule is used to generate a fused data set based on a plurality of the image data packets.

[0019] Furthermore, the step of extracting image information from the plurality of imaging images to generate image data packets comprises:

[0020] Based on the preset grid specifications of the grid cells, construct an average gradient grid matrix;

[0021] If the image is a color image, converting the image into a grayscale image;

[0022] Preprocessing the grayscale image or the imaged image, and using a gradient operator to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaged image;

[0023] Calculating the average gradient value of the pixel points corresponding to the grid unit, storing each of the average gradient values ​​into the average gradient grid matrix, and generating an image grid matrix;

[0024] Convert the imaged image into a normalized floating point image, construct a first pyramid by a down sampling method based on a preset number of pyramid layers, and construct a second pyramid by an up sampling method based on the first pyramid;

[0025] The imaged image, the image grid matrix and the second pyramid are stored in an image data packet.

[0026] Furthermore, the streaming fusion method performs fusion processing on the fusion data set to obtain a fusion result image, including:

[0027] For N continuously received imaging images, based on a preset number of fusion data blocks n, each time fusion occurs, image data packets of n imaging images are selected from the fusion data set for image fusion to obtain a fused image data packet at each fusion event;

[0028] The fused image data packet generated during each fusion is stored in the fused data set, and the corresponding fused image data packet is used as a fused data block during the next fusion for secondary fusion;

[0029] Through multiple iterations of fusion, until the image data packets of N imaging images have completed at least one fusion, the fusion result image is output;

[0030] The number of fused data blocks is the number of image data packets when performing one image fusion.

[0031] Furthermore, the image data packets of n imaging images are selected from the fusion data set for image fusion to obtain a fused image data packet at each fusion, including

[0032] For n imaging images, traverse the image grid matrix of n image data packets;

[0033] Comparing the average gradient values ​​of the grid cells at corresponding positions of the n image grid matrices, and selecting the grid cell with the largest average gradient value as the image fusion area;

[0034] Based on the image fusion area, constructing corresponding mask images for the n image grid matrices;

[0035] Traversing the second pyramids of the n image data packets, performing pixel value fusion processing on the pixel values ​​of the n imaging images in the image fusion area in the mask image, and obtaining a pyramid fusion image subset;

[0036] Performing upward recovery sampling on the pyramid fused image subset to obtain a fused image;

[0037] Based on the fused image, image information is extracted through a pre-preparation submodule to obtain a fused image data packet.

[0038] Further, the external interaction unit includes an interface module, an interaction logic module and a storage module;

[0039] The interface module is used to provide an interactive interface for the user to perform interactive actions, wherein the interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode, wherein the imaging fusion mode includes a manual fusion mode, an automatic fusion mode, and a custom fusion mode;

[0040] The interactive logic module is used to convert the interactive action into an interactive instruction, and retrieve the image information from the storage module to the interface module for software display;

[0041] The storage module is used to store the fusion configuration parameters corresponding to the interaction instruction, and receive image information and imaging focus position, wherein the image information includes an imaging image and a fusion result image.

[0042] Further, the image source unit includes a zoom module and an imaging module;

[0043] The zoom module is used to adjust the imaging focus position based on the imaging fusion mode, the imaging focus position is the relative physical position of the object plane and the imaging plane, and the adjusting of the imaging focus position includes adjusting the position of the object plane or the imaging plane and adjusting the position of the object plane and the imaging plane;

[0044] The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

[0045] A multi-mode imaging fusion device is applied to the multi-mode imaging fusion system.

[0046] A multi-mode imaging fusion method is applied to the above multi-mode imaging fusion system, the system includes an image source unit, a fusion processing unit, an external interaction unit and an image output unit, and the method specifically includes:

[0047] Acquire an imaging fusion mode through the external interaction unit, and feed the imaging fusion mode back to the fusion processing unit;

[0048] The image source unit acquires the imaging fusion mode through the fusion processing unit, adjusts the imaging focus position based on the imaging fusion mode, generates a plurality of imaging images based on the imaging focus position, and feeds the imaging images back to the fusion processing unit;

[0049] Feeding back the imaging fusion mode to the image source unit through the fusion processing unit, extracting image information from the plurality of imaging images respectively to generate a fusion data set, performing fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feeding back the fusion result image to the image output unit;

[0050] The fusion result image is received through the image output unit, and the fusion result image is visualized.

[0051] Furthermore, the process of extracting image information from the plurality of imaging images is an asynchronous processing process.

[0052] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0053] A multi-mode imaging fusion system of the present application extracts the image information of each imaging image before image fusion processing, that is, performs a pre-preparation operation to ensure that the effective information of each image can be fully utilized in the subsequent fusion process, thereby improving the accuracy and quality of the fusion result. Then, in the fusion process, a streaming fusion method is used to perform real-time fusion of the data of each extracted image data packet, so that the imaging image can be processed in real time without waiting for all data to be collected before fusion, thereby improving processing efficiency. In addition, the fusion process of the present application can adapt to fusion implementation methods in various scenarios, that is, the image fusion method in different imaging fusion modes is performed in the fusion processing module through pre-preparation operations and streaming fusion methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 is a schematic diagram of the optical principle of the depth of field effect provided by an embodiment of the present invention;

[0056] Figure 2is a structural schematic diagram of a multi-mode imaging fusion system provided by an embodiment of the present invention;

[0057] Figure 3 It is a flowchart of a pre-preparation operation provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic flow chart of a process for fusing two image data packets provided by an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of a process of performing imaging and image fusion in different imaging fusion modes provided by an embodiment of the present invention;

[0060] Figure 6 It is a structural schematic diagram of a feasible embodiment of adjusting the rear-end optical component provided by the present invention;

[0061] Figure 7 It is a flowchart of a multi-mode imaging fusion method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0063] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0064] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0066] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0068] See also Figure 1-6 As shown, the present invention is a multi-mode imaging fusion system, comprising: an image source unit, a fusion processing unit, an external interaction unit and an image output unit;

[0069] The external interaction unit is used to obtain an imaging fusion mode and feed the imaging fusion mode back to the fusion processing unit;

[0070] The image source unit is used to obtain the imaging fusion mode through the fusion processing unit, adjust the imaging focus position based on the imaging fusion mode, generate a plurality of imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit;

[0071] The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, and extract image information from the plurality of imaging images to generate a fusion data set, adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed back the fusion result image to the image output unit;

[0072] The image output unit is used to receive the fusion result image and visualize the fusion result image.

[0073] The multi-mode imaging fusion system of the present application is based on the optical principle of depth of field effect, specifically, Figure 1As shown, there are points B1 and B2 on the object plane A1 and the object plane A2, respectively. Point B1 and point B2 are imaged as B1' and B2' on the imaging planes A1' and A2', respectively. However, when imaging is performed by an imaging device such as a camera sensor, it is usually performed based on the reference image plane A'. On the reference image plane A', the images corresponding to B1' and B2' are diffuse spots shown as Z'. In a feasible embodiment, when the size of the diffuse spot on the reference image plane A' is not perceived by the human eye, the objects on the object plane A1 and the object plane A2 are clearly visible on the reference image plane A'. It can be understood that when the size of the light spot imaged on the reference image plane A' is not perceived by the human eye within the depth of field, the objects within the depth of field are clearly visible on the imaging device. Therefore, in the present application, by controlling the relative movement of the object plane A and the reference image plane A', an image set with a depth of field range exceeding the original optical path is obtained, and then a clear image with a larger depth of field range is synthesized by performing a fusion operation on the image set.

[0074] In the present application, the imaging focus position is adjusted according to the imaging fusion mode. It can be understood that there are multiple imaging fusion modes. Based on different imaging fusion modes, the imaging focus position is adjusted in different ways to adapt to the user's use in multiple scenarios, wherein the imaging focus position is the relative physical position of the object plane and the imaging plane, that is, the relative position of the above-mentioned control object plane A and the reference image plane A'. By acquiring multiple imaging images generated in the process of adjusting the imaging focus position, and extracting image information from the multiple imaging images to generate a fused data set, a streaming fusion method is used to fuse the fused data set to obtain a fused result image. It can be understood that in the present application, when performing Before the image fusion process, the image information of each imaging image is extracted first to ensure that the effective information of each image can be fully utilized in the subsequent fusion process to improve the accuracy and quality of the fusion result. Then, in the fusion process, the streaming fusion method is used to perform real-time fusion of the data of each extracted image data packet, so that the imaging image can be processed in real time without waiting for all the data to be collected before fusion, thereby improving the processing efficiency. In addition, the fusion process of the present application can adapt to the fusion implementation methods in various scenarios, that is, the image fusion method in different imaging fusion modes is performed in the fusion processing module through pre-preparation operations and streaming fusion methods.

[0075] In some of the embodiments, the fusion processing unit includes a fusion logic module, an image buffer module and a fusion processing module;

[0076] The fusion logic module is used to receive and feed back the interaction instruction of the external interaction unit to the image cache module and the image source unit;

[0077] The image cache module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module;

[0078] The fusion processing module is used to receive the fusion data set, and adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed the fusion result image back to the image output unit and the external interaction unit.

[0079] When the multi-mode imaging fusion system is applied to a multi-mode imaging fusion device, the fusion processing unit is a Linux program running on a DSP chip, and is not limited by the physical device on which it is carried and the operating system it runs.

[0080] Specifically, Figure 2 As shown, the fusion logic module receives the interaction instructions fed back by the external interaction unit and the fusion configuration parameters corresponding to the interaction instructions, and feeds back the relevant interaction instructions to the image source unit, so that the image source unit completes the imaging of the source image according to the interaction instructions and generates an imaging image. The generated imaging image and source image are output to the image cache module for image information extraction and image data packet caching, and the fusion data set in the image cache module is output to the fusion processing module for fusion processing, and finally a fusion result image is generated, and the generated fusion result image is fed back to the external interaction unit and the image output unit, so as to realize the visualization of the fusion result image.

[0081] In this embodiment, the feedback modes of the fusion logic module include passive reply feedback and active upload feedback, and the two feedback modes cooperate with each other to realize information interaction between the fusion processing unit, the image source unit and the external interaction unit.

[0082] In some of the embodiments, the image cache module includes a data receiving submodule, a pre-preparation submodule and a data collection submodule;

[0083] The data receiving submodule is used for receiving the source image and the imaging image;

[0084] The pre-preparation submodule is used to extract image information from the plurality of imaging images respectively to generate image data packets;

[0085] The data set submodule is used to generate a fused data set based on a plurality of the image data packets.

[0086] In this embodiment, the image cache module includes a pre-preparation operation for each imaging image before fusion, that is, image information is extracted from multiple imaging images respectively, and finally packaged into an image data packet. It can be understood that the image data packet is other image data and parameter data generated by the imaging image through image processing. By separating the image processing operation before the imaging image fusion processing to the image cache unit, the image information extraction of the imaging image and the asynchronous operation between the fusion of each imaging image are realized, and then the image fusion processing process will not block the image cache module from continuously receiving the imaging image, thereby realizing the real-time output of the fused data. Therefore, the image cache module includes a data receiving submodule, a pre-preparation submodule and a data collection submodule. Through the cooperation of the above three submodules, the pre-preparation operation of the imaging image and the caching of the corresponding image information are realized.

[0087] In this embodiment, the number of image data packets in the input fusion data set can be a fixed value of 2, or can be any value greater than 2. When the number of image data packets in the input fusion data set is any value greater than 2, the image data packets generated after image information extraction of each imaging image can be stored in the fusion data set. During the fusion process, streaming fusion is performed by calling related image data packets from the fusion data set, thereby realizing image data fusion of multiple imaging images.

[0088] In this embodiment, when the image storage module obtains the imaging image fed back by the image source module, the pre-fusion preparatory operation is synchronously performed on the imaging image, and the pre-fusion preparatory operation of different imaging images in the image storage module is an asynchronous operation. When the previous pre-fusion preparatory operation is not completed and a new imaging image is input, it is processed by creating a new thread without waiting for the previous pre-fusion preparatory operation to be completed, thereby improving processing efficiency.

[0089] In some of these embodiments, the image data packet includes an imaged image, an image grid matrix, and a second pyramid;

[0090] The step of extracting image information from the plurality of imaging images to generate image data packets comprises:

[0091] Based on the preset grid specifications of the grid cells, construct an average gradient grid matrix;

[0092] If the image is a color image, converting the image into a grayscale image;

[0093] Preprocessing the grayscale image or the imaged image, and using a gradient operator to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaged image;

[0094] Calculating the average gradient value of the pixel points corresponding to the grid unit, storing each of the average gradient values ​​into the average gradient grid matrix, and generating an image grid matrix;

[0095] Convert the imaged image into a normalized floating point image, construct a first pyramid by a down sampling method based on a preset number of pyramid layers, and construct a second pyramid by an up sampling method based on the first pyramid;

[0096] The imaged image, the image grid matrix and the second pyramid are stored in an image data packet.

[0097] In this embodiment, when extracting image information from an imaged image, if the imaged image is a color image, it is necessary to first convert the imaged image into a grayscale image, and then construct an image grid matrix, thereby reducing the data dimension, thereby significantly reducing the computational complexity and time cost of subsequent image information extraction.

[0098] In this embodiment, the average gradient value of the imaging image is combined with the grid matrix to construct an average gradient grid matrix for describing the gradient change relationship between pixels in the imaging image. In addition, the imaging image is down-sampled and up-sampled to construct details of the imaging image and image reconstruction, wherein the average gradient grid matrix and the second pyramid are the data basis for the fusion of multiple imaging images.

[0099] Specifically, for the average gradient grid matrix, before construction, the grid specifications of the grid units need to be preset according to the resolution and characteristic scale of the imaging image. In a preferred embodiment, the specification of a single grid unit is 100×100 pixels, that is, the image is divided into multiple grid units of 100×100 pixels. For an imaging image of size 1920×1080, the size specification of the finally generated average gradient grid matrix is ​​20×11, that is, 1920 / 100 and 1080 / 100 are rounded up to obtain; in addition, for the second pyramid, before construction, the number of pyramid layers needs to be preset according to the size requirement of the target layer image. In a preferred embodiment, the number of pyramid layers is 3.

[0100] Specifically, Figure 3As shown, when the image grid matrix is ​​generated by the average gradient grid matrix, the grayscale image or the imaging image is preprocessed, such as performing Gaussian filtering on the grayscale image or the imaging image to reduce noise and interference, and then the gradient value of each pixel in the preprocessed grayscale image or the imaging image is calculated. Specifically, the gradient value of each pixel can be obtained by using a gradient operator, such as a Sobel operator or a Prewitt operator. For each grid unit, the corresponding average gradient value is calculated, and the corresponding average gradient value is filled into the grid matrix, so that each element of the average gradient grid matrix represents the gradient change value between adjacent pixels, thereby generating the image grid matrix.

[0101] In this embodiment, Figure 3 As shown, when constructing the second pyramid, the imaging image is converted into a normalized floating-point image, specifically, the pixel values ​​of the imaging image are converted into floating-point numbers between 0 and 1. Specifically, the maximum and minimum values ​​of the pixel values ​​in the imaging image are calculated, and each pixel value is converted into a floating-point number between 0 and 1 based on a conversion formula. In a feasible embodiment, the conversion formula is: (pixel value-minimum value) / (maximum value-minimum value).

[0102] In this embodiment, the first pyramid constructed by the down-sampling method is a Gaussian image pyramid, which is used for down-sampling and scale space representation of an image. Specifically, the size representation of each layer of the pyramid is obtained by filtering the previous layer of image using a Gaussian blur kernel, thereby generating a first pyramid. Then, based on the first pyramid, an upward sampling method is used to construct a second pyramid, wherein the second pyramid is a Laplacian image pyramid, which is constructed by taking the difference between an image of each layer in the Gaussian image pyramid and an image of the previous layer, and is mainly used for detail extraction and reconstruction of an imaged image.

[0103] It can be understood that in this embodiment, the construction of the image grid matrix and the second pyramid can be carried out simultaneously, so as to make full use of computing resources and reduce the overall processing time, and then stored in the image data packet, wherein the image data packet also includes the unprocessed imaging image, so as to facilitate the rapid access to relevant data in the subsequent fusion process.

[0104] In some embodiments, the streaming fusion method performs fusion processing on the fusion data set to obtain a fusion result image, including:

[0105] For N continuously received imaging images, based on a preset number of fusion data blocks n, each time fusion occurs, image data packets of n imaging images are selected from the fusion data set for image fusion to obtain a fused image data packet at each fusion event;

[0106] The fused image data packet generated during each fusion is stored in the fused data set, and the corresponding fused image data packet is used as a fused data block during the next fusion for secondary fusion;

[0107] Through multiple iterations of fusion, until the image data packets of N imaging images have completed at least one fusion, the fusion result image is output;

[0108] The number of fused data blocks is the number of image data packets when performing one image fusion.

[0109] In this embodiment, N continuously received imaging images are marked as ImageInput1, ImageInput2, ..., ImageInputN, and then preparatory actions are performed on them respectively to generate image data packets data1, data2, ..., dataN. The fused image data packets generated during each fusion are marked as g_data1, g_data2, ..., and the corresponding fused image data packets are stored in a fused data set for use as a fused data block during the next fusion.

[0110] Since the number of image data packets in the input fusion data set can be a fixed value of 2 or any value greater than 2, when the number of image data blocks in the fusion data set is a fixed value of 2, one of the data in the fusion data set is an empty fusion image data packet g_data1, and during the fusion process, the fusion image data packet g_data1 is iteratively assigned to g_data2 or g_data3, etc. Based on this, when the number of input fusion data sets is fixed to 2, the problem of excessive time consumption caused by the pre-preparation operation of multiple cycles of fusion image data packets can be avoided. When the number of input fusion data sets is greater than 2, since the pre-preparation operation of each imaging image has been completed through the image cache unit and cached in the fusion data set, the image data blocks can be further fused accordingly to further improve the processing speed.

[0111] In this embodiment, before data fusion is performed, the number of fused data blocks is determined. Reasonable setting of the number of fused data blocks can effectively improve the real-time performance of the fusion of each imaging image. In some preferred embodiments, the number of fused data blocks is 2.

[0112] In this embodiment, a specific description is given by taking the number of fused data blocks as 2 and the number of image data packets in the input fused data set as a fixed value of 2 as an example. Specifically, data1 of ImageInput1 received in the fused data set is assigned to the above-mentioned empty fused data unit g_data1, and data2 of the next imaging image ImageInput2 is input into the fused data set for fusion. It can be understood that only one imaging image cannot be fused, so data1 of the first imaging image ImageInput1 is used as the fused image data packet g_data1 in the fused data set, and data2 of the next imaging image ImageInput2 is waited for for image fusion to generate a new fused image data packet g_data2. That is, g_data2=g_data1+data2, wherein g_data2 is a fused data block for the next fusion. When data3 of the third imaging image ImageInput3 is input into the fused data set, g_data2 and data3 are fused to generate a fused image data packet g_data3. At this time, g_data3=g_data2+data3. In this streaming fusion manner, each image data packet in the input fused data set is iteratively fused until all imaging images have completed the corresponding fusion operation. At this time, the fused image corresponding to g_dataN in the last fusion operation g_dataN=g_data(N-1)+dataN is fed back to the image output unit and the external interaction unit as the fusion result image.

[0113] In addition, for an embodiment in which the number of fused data blocks is a value greater than 2, and the number of image data packets in the input fused data set is a fixed value of 2, a feasible implementation method is: the image data blocks that do not meet the fusion requirements are stored in the fused image data packet g_data1 for fusion preparation, and when the last fused data block is input into the fused data set, the data fusion operation is performed. Specifically, when the number of fused data blocks is 3, data1 and data2 of the first imaging image and the second imaging image are stored in the fused image data packet g_data1, and when data3 of the third imaging image is input into the fused data set, data fusion is performed on data1, data2 and data3 to generate a fused image data packet g_data2, and iteratively assigned to g_data1, waiting for the input of the image data packet of the next imaging image.

[0114] In some embodiments, the image data packets of n imaging images are selected from the fusion data set for image fusion to obtain a fused image data packet at each fusion, including:

[0115] For n imaging images, traverse the image grid matrix of n image data packets;

[0116] Comparing the average gradient values ​​of the grid cells at corresponding positions of the n image grid matrices, and selecting the grid cell with the largest average gradient value as the image fusion area;

[0117] Based on the image fusion area, constructing corresponding mask images for the n image grid matrices;

[0118] Traversing the second pyramids of the n image data packets, performing pixel value fusion processing on the pixel values ​​of the n imaging images in the image fusion area in the mask image, and obtaining a pyramid fusion image subset;

[0119] Performing upward recovery sampling on the pyramid fused image subset to obtain a fused image;

[0120] Based on the fused image, image information is extracted through a pre-preparation submodule to obtain a fused image data packet.

[0121] In this embodiment, the image fusion process is specifically described by taking the number of fusion data blocks as 2 as an example. Specifically, the image grid matrices in the two image data packets are traversed respectively, and the average gradient values ​​of the two corresponding grid units in the two image grid matrices are compared, and the grid unit with the largest average gradient value is selected as the image fusion area. It can be understood that the grid unit with the largest average gradient value usually contains more image details and features, so it is selected as a potential fusion area for subsequent image fusion. Further, for the two image grid matrices, according to the image fusion area corresponding to the corresponding image grid matrix, a corresponding mask image is constructed for quantitative marking of the image fusion area. On the basis of the mask image, the second pyramid in the two image data packets is traversed, and the pixel values ​​of the parts corresponding to the image fusion area in the two imaging images are fused to obtain a pyramid fusion image subset, and the pyramid fusion image subset is upwardly restored and sampled, so that a fusion image generated by the fusion of the two imaging images can be obtained. For non-last iterative fusion, the image information of the fusion image is extracted by the pre-preparation submodule to generate a fusion image data packet, and the fusion image is stored in the fusion data set, and is fused twice with the next image data packet. For the last iterative fusion, the fusion image is the fusion result image.

[0122] In some embodiments, the method of fusing pixel values ​​of a portion of the imaging image corresponding to the image fusion area includes but is not limited to using a weighted fusion method.

[0123] Specifically, Figure 4As shown, two image data packets are marked as A and B respectively, then the average gradient values ​​of the grid cells at the corresponding positions of the two image grid matrices are compared, the grid cell with the largest average gradient value is selected as the image fusion area, and multiple image fusion areas belonging to the same image network matrix are integrated into a mask image, specifically including: constructing corresponding index image matrices for the image data packets A and B of the two imaging images respectively, wherein the index image matrix is ​​adapted to the image grid matrix of the imaging image, and configuring a first pixel value, specifically 0, for each grid cell in the index image matrix; comparing the average gradient values ​​of the corresponding grid cells of the image grid matrices in the two image data packets, and selecting the grid cell with the largest average gradient value as the image fusion area domain, and modify the first pixel value of the grid unit corresponding to the image fusion area in the index image matrix to the second pixel value, specifically 1. At the same time, a mask image is constructed based on the index image matrix. Specifically, for the grid unit with a pixel value of 1 in the index image matrix, all pixel values ​​of the grid unit at the corresponding position on the mask image are set to 1; thereafter, the second pyramids of the image data packet A and the image data packet B are traversed respectively, and the pixel fusion of the imaging image is performed on the corresponding pyramid level for the part with a pixel value of 1 in the mask image. Preferably, the pixel fusion is performed in a pixel value weighted average manner to obtain a fused pyramid fusion image subset, and upward recovery sampling is performed on the basis of the pyramid fusion image subset to obtain a fused image.

[0124] In some of the embodiments, the external interaction unit includes an interface module, an interaction logic module, and a storage module;

[0125] The interface module is used to provide an interactive interface for the user to perform interactive actions, wherein the interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode, wherein the imaging fusion mode includes a manual fusion mode, an automatic fusion mode, and a custom fusion mode;

[0126] The interactive logic module is used to convert the interactive action into an interactive instruction, and retrieve the image information from the storage module to the interface module for software display;

[0127] The storage module is used to store the fusion configuration parameters corresponding to the interaction instruction, and receive image information and imaging focus position, wherein the image information includes an imaging image and a fusion result image.

[0128] When the above-mentioned multi-mode imaging fusion system is applied to a multi-mode imaging fusion device, the external interaction unit is a Linux program running on the DSP chip, which is not limited by the physical device it carries and the operating system it runs. It can be an Android program on a mobile device, an IOS program on an IOS device, an exe program on a windows system device, or a supported browser web program, and communicates with the fusion processing unit, and the communication method is not limited to a specific protocol method. Specifically, the communication content between the external interaction unit and the fusion processing unit includes but is not limited to fusion mode switching information, fusion area configuration information, fusion process control information, fusion intermediate image feedback information, and process operation feedback information.

[0129] In this embodiment, the user can select the fusion processing mode and configure the fusion configuration parameters under the corresponding fusion processing mode through the interface unit, such as the zoom range, zoom amount, zoom speed, fusion weight, specified fusion area position information, process node jump configuration, etc. during fusion, and the storage unit is responsible for storing the fusion configuration parameters corresponding to the interactive instructions, receiving the imaging image, receiving the fusion result screen, receiving the focus position information, etc. The interactive logic unit is responsible for converting the user's interactive actions into interactive instructions, starting the corresponding operation process, and retrieving the imaging image and fusion result image of the storage unit to the interface unit for software display.

[0130] In the present embodiment, the mode of controlling the relative motion between the object plane A and the reference image plane A', that is, the imaging fusion mode, has three modes, namely, a manual fusion mode, an automatic fusion mode and a custom fusion mode, wherein the custom focus range fusion is an implementation set under different focus ranges, such as a focus range consisting of 1 / 3 focus range, 2 / 3 focus range or a focus range consisting of different focus start and end positions. Based on the above three imaging fusion modes, the fusion process of the present application can adapt to different fusion implementation mechanisms in a variety of scenarios, that is, the above three imaging fusion modes can all use the pre-preparation operation of the imaging image in the above-mentioned fusion processing unit and the streaming fusion method to perform image fusion.

[0131] Specifically, Figure 5As shown, in the manual fusion mode, the interface controls on the interactive interface provided by the external interaction unit can be used for adjustment, such as the position slider to directly adjust the imaging focus position in the image source unit, and then output the corresponding imaging image, so as to perform real-time fusion output, that is, in the manual fusion mode, the corresponding interaction instructions need to be provided by the external interaction unit. In particular, after the imaging focus position is adjusted through the manual fusion mode, the fusion logic unit of the fusion processing unit starts the stop timing action, and stops the fusion after the timing timeout, thereby reducing the system load pressure. If there is a new zoom adjustment during the timing process, the timing action is canceled. In the automatic fusion mode, image clarity detection is provided. When it is detected that the rising trend of clarity disappears continuously, the fusion process is automatically stopped. In one embodiment, the sum of the Sobel gradient values ​​in the horizontal and vertical directions is used as the clarity value. When it is detected that the rising trend of clarity disappears continuously for 5 times, the fusion process is automatically stopped. In the customized focus range fusion mode, the fusion logic unit of the fusion processing unit is responsible for controlling the zoom module according to the preset range or step value to achieve focus fusion within the focus range. Stopping the fusion action is only triggered when the focus within the focus range is completed.

[0132] In some of the embodiments, the image source unit includes a zoom module and an imaging module;

[0133] The zoom module is used to adjust the imaging focus position based on the imaging fusion mode, the imaging focus position is the relative physical position of the object plane and the imaging plane, and the adjusting of the imaging focus position includes adjusting the position of the object plane or the imaging plane and adjusting the position of the object plane and the imaging plane;

[0134] The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

[0135] In this embodiment, the imaging focus position is adjusted by the zoom module, and the imaging focus position corresponds to Figure 1 The relative movement between the controlled object plane A and the reference image plane A' shown is specifically the relative physical position of the photographed object plane and the imaging plane. In addition, the source image is converted into an imaging image through the imaging module and output to the fusion processing unit for image fusion processing. It can be understood that the output format of the imaging image is a digital image signal, which is conducive to image processing.

[0136] Specifically, the zoom unit is an external optical path device acting on the imaging optical path, and the imaging unit is a photoelectric device used to obtain the final imaging of the optical path. When the above-mentioned multi-mode imaging fusion system is applied to the multi-mode imaging fusion device, the zoom unit can be a motor focusing device, and the imaging unit can be a global shutter sensor, a linear sensor or a rolling shutter camera, etc., and is not limited to the specific form type of the physical device.

[0137] In some embodiments, adjusting the imaging focus position may specifically be adjusting Figure 1 The object plane or imaging plane shown in the figure, the above-mentioned external optical path device includes a front-end optical component, a rear-end optical component and a driving component, wherein the front-end optical component is usually arranged at the front end of the optical path imaging device, and is responsible for receiving the light of the object, that is, the light signal, and the front-end optical component includes but is not limited to a lens and a compensation lens. In a feasible embodiment, the supplementary lens is connected to the driving component for driving, and the step value and the step number are set for the driving component, so that the supplementary lens moves during the focusing process, and then the front-end optical component is adjusted to achieve the change of the imaging focus position. In another feasible embodiment, the movement of the supplementary lens does not depend on The driving component, that is, through artificial control of the connection structure associated with the supplementary lens, makes the supplementary lens move, thereby achieving focusing. Specifically, it can be adapted to the imaging fusion mode of the manual fusion mode. When in the manual fusion mode, the trigger timer is set by the external interaction unit, and a fusion control instruction is sent to the fusion logic module of the fusion processing unit at a specified time interval. The fusion logic module then sends an instruction containing only the trigger information to the zoom module of the image source unit. At this time, the zoom module only serves to trigger the imaging module to output the image. The output of the source image is ultimately started and stopped by the external interaction unit, and the output process is controlled by the timer of the external interaction unit.

[0138] The back-end optical component converts the optical signal captured by the front-end optical component into an electrical signal to form a digital image. The back-end optical component includes but is not limited to an image sensor and an optical interface, wherein the optical interface is used to connect the back-end optical component and the front-end optical component. In a specific application, the optical interface may be a C interface or other interface, which is used to fix the back-end optical component to the lens in the optical imaging device or to embed the back-end optical component as a submodule in the eyepiece camera in the optical imaging device. In addition, in a feasible embodiment, the image sensor is connected to the driving component to adjust the plane where the image sensor is located to achieve the movement of the imaging plane. Specifically, as Figure 6As shown, the reference focus point P1' corresponding to the object point P1 is between the imaging planes A1'~A2'. When the driving component moves, the image sensor plane is displaced equally under the action of the driving component, thereby obtaining imaging data passing through the P1' plane. In the fusion process, the image gradient detection action can retain the detail information of P1' in the picture. Similarly, any point located in the planes A1~A2 of the photographed object can also be processed as described above to form a clear fusion result image of the imaging range A1'~A2' of the A1~A2 planes. In another feasible embodiment, the movement of the image sensor does not rely on the driving component, that is, the connection structure associated with the image sensor is manually controlled. The specific implementation process is similar to the way in which the connection structure associated with the supplementary lens is manually controlled in the front-end optical component to adjust the focus, which will not be repeated here.

[0139] The driving component can be specifically a device such as a motor that can drive the movement of the supplementary lens or image sensor. Its specific structure will not be described in detail here. The two driving modes of controlling the movement of the supplementary lens or image sensor through the driving component and manually controlling the movement of the supplementary lens or image sensor make the multi-mode imaging fusion system of the present application compatible with optical imaging devices of different costs, providing a low-cost expansion method. The lens used in the optical imaging device can be a non-fixed focus lens, a ZOOM lens or an AF lens. The optical imaging device can also be a microscope or telescope with an automatic focusing function. Therefore, the multi-mode fusion imaging system has good expansion adaptability. It can be understood that the optical imaging device is a multi-mode imaging fusion device that uses the multi-mode imaging fusion system.

[0140] Specifically, when adjusting the imaging focus position, it can be achieved by adjusting the front-end optical component or adjusting the rear-end optical component or adjusting both the front-end optical component and the rear-end optical component. It can be specifically understood that the final manifestation of the imaging focus position adjustment process of the zoom module is the physical displacement of the subject plane and the imaging plane, wherein the adjustment of the subject plane is achieved by adjusting the front-end optical component, and the adjustment of the imaging plane is achieved by adjusting the rear-end optical component.

[0141] In the present invention, the zoom unit is not limited to the actual adjustment position of the optical path of the optical component, nor is it limited to the specific method of combining multiple optical components to achieve zoom adjustment. The above-mentioned front-end optical component and rear-end optical component are a description of an embodiment of zoom adjustment, rather than a limitation thereto.

[0142] The image output unit specifically includes a screen display module, which is responsible for directly displaying the imaging image and presenting the fusion result image in the form of a floating window, an embedded window, a switching page window, multiple windows, etc.

[0143] In the present application, a multi-mode imaging fusion device is also provided, comprising the multi-mode imaging fusion system mentioned above.

[0144] Specifically, the multi-mode imaging fusion device includes a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, the multi-mode imaging fusion method can be implemented. The multi-mode imaging fusion device can be an optical imaging device with a focusing imaging function such as a microscope or a telescope. By applying the multi-mode imaging fusion system to the multi-mode imaging fusion device, automatic focusing imaging of devices such as microscopes or telescopes can be implemented.

[0145] See also Figure 7 As shown, the present application also provides a multi-mode imaging fusion method, which is applied to the above-mentioned multi-mode imaging fusion system, wherein the system includes an image source unit, a fusion processing unit, an external interaction unit and an image output unit, and the method specifically includes:

[0146] S100, acquiring an imaging fusion mode through the external interaction unit, and feeding back the imaging fusion mode to the fusion processing unit;

[0147] S200, enabling the image source unit to acquire the imaging fusion mode through the fusion processing unit, and adjusting the imaging focus position based on the imaging fusion mode, generating a plurality of imaging images based on the imaging focus position, and feeding the imaging images back to the fusion processing unit;

[0148] S300, feeding back the imaging fusion mode to the image source unit through the fusion processing unit, extracting image information from the plurality of imaging images respectively to generate a fusion data set, performing fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feeding back the fusion result image to the image output unit;

[0149] S400: Receive the fusion result image through the image output unit, and visualize the fusion result image.

[0150] In the present application, the imaging fusion mode and the corresponding fusion configuration parameters are obtained through the interface module of the external interaction unit, the interactive logic module converts the imaging fusion mode into an interactive instruction and feeds it back to the fusion logic module of the fusion processing unit, the fusion logic module feeds back the interactive instruction to the zoom module of the image source unit, the zoom module adjusts the imaging focus position according to the imaging fusion mode, and then converts the source image into an imaging image based on the imaging focus position through the imaging module of the image source unit, and further feeds back the imaging image to the image cache module of the fusion processing unit, the image cache module extracts image information from the imaging image and finally generates a fusion data set, the image cache module outputs the fusion data set to the fusion processing module for image fusion processing to generate a fusion result image, and the finally generated fusion result image is output to the image output unit for visual presentation, in addition, the fusion process of each imaging image is an independent process, and the imaging image is always cached during the fusion process, specifically, the fusion result image and the imaging image are also output to the storage module of the external interaction unit for storage, so as to be retrieved to the interface unit for software display.

[0151] In one embodiment, the process of extracting image information from the plurality of imaging images is an asynchronous processing process.

[0152] In this embodiment, the image information extraction of the imaging image and the fusion of different imaging images are asynchronous operations, so that the image fusion processing process cannot block the image cache module from continuously receiving the imaging image. In addition, the pre-preparation operation of different imaging images in the image storage module is also an asynchronous operation. When the previous pre-preparation operation is not completed and a new imaging image is input, it is processed by creating a new thread without waiting for the previous pre-preparation operation to be completed, thereby improving processing efficiency.

[0153] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-mode imaging fusion system, characterized in that: include: Image source unit, fusion processing unit, external interaction unit and image output unit; The external interaction unit is used to obtain an imaging fusion mode and feed the imaging fusion mode back to the fusion processing unit; The image source unit is used to obtain the imaging fusion mode through the fusion processing unit, adjust the imaging focus position based on the imaging fusion mode, generate a plurality of imaging images based on the imaging focus position, and feed the imaging images back to the fusion processing unit; The fusion processing unit is used to feed back the imaging fusion mode to the image source unit, and extract image information from the plurality of imaging images to generate a fusion data set, adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed back the fusion result image to the image output unit; The image output unit is used to receive the fusion result image and visualize the fusion result image.

2. The system according to claim 1, characterized in that The fusion processing unit includes a fusion logic module, an image buffer module and a fusion processing module; The fusion logic module is used to receive and feed back the interaction instruction of the external interaction unit to the image cache module and the image source unit; The image cache module is used to cache image data packets of multiple imaging images to generate a fused data set, and feed the fused data set back to the fusion processing module; The fusion processing module is used to receive the fusion data set, and adopt a streaming fusion method to perform fusion processing on the fusion data set to obtain a fusion result image, and feed the fusion result image back to the image output unit and the external interaction unit.

3. The system according to claim 2, characterized in that The image cache module includes a data receiving submodule, a pre-preparation submodule and a data collection submodule; The data receiving submodule is used for receiving the source image and the imaging image; The pre-preparation submodule is used to extract image information from the plurality of imaging images respectively to generate image data packets, wherein the image data packets include imaging images, image grid matrices and second pyramids; The data set submodule is used to generate a fused data set based on a plurality of the image data packets.

4. The system according to claim 3, characterized in that The step of extracting image information from the plurality of imaging images to generate image data packets comprises: Based on the preset grid specifications of the grid cells, construct an average gradient grid matrix; If the image is a color image, converting the image into a grayscale image; Preprocessing the grayscale image or the imaged image, and using a gradient operator to calculate the gradient value of each pixel in the preprocessed grayscale image or the imaged image; Calculating the average gradient value of the pixel points corresponding to the grid unit, storing each of the average gradient values ​​into the average gradient grid matrix, and generating an image grid matrix; Convert the imaged image into a normalized floating point image, construct a first pyramid by a down sampling method based on a preset number of pyramid layers, and construct a second pyramid by an up sampling method based on the first pyramid; The imaged image, the image grid matrix and the second pyramid are stored in an image data packet.

5. The system according to claim 3, characterized in that The streaming fusion method performs fusion processing on the fusion data set to obtain a fusion result image, including: For N continuously received imaging images, based on a preset number of fusion data blocks n, each time fusion occurs, image data packets of n imaging images are selected from the fusion data set for image fusion to obtain a fused image data packet at each fusion event; The fused image data packet generated during each fusion is stored in the fused data set, and the corresponding fused image data packet is used as a fused data block during the next fusion for secondary fusion; Through multiple iterations of fusion, until the image data packets of N imaging images have completed at least one fusion, the fusion result image is output; The number of fused data blocks is the number of image data packets when performing one image fusion.

6. The system according to claim 5, characterized in that The step of selecting n image data packets of imaging images from the fusion data set for image fusion to obtain a fused image data packet at each fusion, includes: For n imaging images, traverse the image grid matrix of n image data packets; Comparing the average gradient values ​​of the grid cells at corresponding positions of the n image grid matrices, and selecting the grid cell with the largest average gradient value as the image fusion area; Based on the image fusion area, constructing corresponding mask images for the n image grid matrices; Traversing the second pyramids of the n image data packets, performing pixel value fusion processing on the pixel values ​​of the n imaging images in the image fusion area in the mask image, and obtaining a pyramid fusion image subset; Performing upward recovery sampling on the pyramid fused image subset to obtain a fused image; Based on the fused image, image information is extracted through a pre-preparation submodule to obtain a fused image data packet.

7. The system according to claim 1, characterized in that The external interaction unit includes an interface module, an interaction logic module and a storage module; The interface module is used to provide an interactive interface for the user to perform interactive actions, wherein the interactive actions include selecting an imaging fusion mode and selecting fusion configuration parameters corresponding to the imaging fusion mode, wherein the imaging fusion mode includes a manual fusion mode, an automatic fusion mode, and a custom fusion mode; The interactive logic module is used to convert the interactive action into an interactive instruction, and retrieve the image information from the storage module to the interface module for software display; The storage module is used to store the fusion configuration parameters corresponding to the interaction instruction, and receive image information and imaging focus position, wherein the image information includes an imaging image and a fusion result image.

8. The system of claim 1, wherein: The image source unit includes a zoom module and an imaging module; The zoom module is used to adjust the imaging focus position based on the imaging fusion mode, the imaging focus position is the relative physical position of the object plane and the imaging plane, and the adjusting of the imaging focus position includes adjusting the position of the object plane or the imaging plane and adjusting the position of the object plane and the imaging plane; The imaging module is used to convert the source image into an imaging image based on the imaging focus position, and output the imaging image to the fusion processing unit.

9. A multi-mode imaging fusion device, characterized in that: Applicable to the multi-mode imaging fusion system described in any one of claims 1-8.

10. A multi-mode imaging fusion method, characterized in that: The multi-mode imaging fusion system applied to any one of claims 1 to 8 comprises an image source unit, a fusion processing unit, an external interaction unit and an image output unit, and the method specifically comprises: Acquire an imaging fusion mode through the external interaction unit, and feed the imaging fusion mode back to the fusion processing unit; The image source unit acquires the imaging fusion mode through the fusion processing unit, adjusts the imaging focus position based on the imaging fusion mode, generates a plurality of imaging images based on the imaging focus position, and feeds the imaging images back to the fusion processing unit; Feeding back the imaging fusion mode to the image source unit through the fusion processing unit, extracting image information from the plurality of imaging images respectively to generate a fusion data set, performing fusion processing on the fusion data set using a streaming fusion method to obtain a fusion result image, and feeding back the fusion result image to the image output unit; receiving the fusion result image through the image output unit, and visually presenting the fusion result image; The process of extracting image information from the plurality of imaging images is an asynchronous processing process.

Images