An image fusion method, device and electronic equipment

Abstract

Embodiments of the present application provide an image fusion method and device, and electronic equipment, and relate to the technical field of image fusion. The method comprises: obtaining a first target image output by a main channel and a second target image output by a secondary channel; the main channel is a pre-designated image acquisition channel for image fusion, and the field of view of the image output by the main channel remains unchanged; the secondary channel is an image acquisition channel other than the main channel; using a target mapping relationship, performing distortion correction processing on the second target image to obtain a corrected second target image; the target mapping relationship is a mapping relationship for correcting the distortion degree of the second target image to the distortion degree of the first target image; and performing image fusion on the first target image and the corrected second target image to obtain a fused image. Through the present solution, image fusion can be achieved while ensuring that the field of view of the image output by the designated channel is not lost.

Description

Technical Field

[0001] This application relates to the field of image fusion technology, and in particular to an image fusion method, apparatus, and electronic device. Background Technology

[0002] The amount of information contained in an image output from a single image acquisition channel is limited, often insufficient for practical applications. Therefore, images from multiple image acquisition channels are typically fused to obtain a fused image containing more information. These image acquisition channels can include infrared, ultraviolet, or visible light image acquisition channels, and so on.

[0003] Since the output image of each of the multiple image acquisition channels will be distorted, in related technologies, the output image of each of the multiple image acquisition channels needs to be corrected for distortion before image fusion. Distortion refers to the degree of distortion of the image formed by an object relative to the object itself, causing image deformation.

[0004] During distortion correction, the field of view of each image acquisition channel is lost. Since users typically do not want the field of view of a specific channel to be lost, how to achieve image fusion without losing the field of view of the specified channel is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide an image fusion method, apparatus, and electronic device to achieve image fusion without loss of field of view for a specified channel. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide an image fusion method, including:

[0007] Acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel;

[0008] Using a target mapping relationship, distortion correction processing is performed on the second target image to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0009] The first target image and the corrected second target image are fused to obtain a fused image.

[0010] Optionally, the target mapping relationship is specifically a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to the target shape;

[0011] The target shape is the shape of the object represented by the image content of the first target image, as displayed in the first target image.

[0012] Optionally, the target mapping relationship is a coordinate mapping relationship generated through a predetermined calibration method;

[0013] The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object.

[0014] The two image sensors include the main channel image sensor and the secondary channel image sensor.

[0015] Optionally, the calibration process for the predetermined calibration method includes:

[0016] Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object;

[0017] The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship;

[0018] The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object.

[0019] Optionally, when the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape;

[0020] The actual shape of the calibration object is the shape obtained by projecting the target image shape using optical inverse projection.

[0021] Optionally, the method for determining the actual shape of the calibration object includes:

[0022] The shape parameters of the target image shape, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target image shape with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object.

[0023] The specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object.

[0024] Optionally, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel;

[0025] Image fusion is performed on the first target image and the corrected second target image to obtain a fused image, including:

[0026] The corrected second target image is used to identify a specified object region to obtain a candidate image corresponding to the specified object region.

[0027] Determine the image to be fused corresponding to the candidate images; wherein, the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate images;

[0028] The image to be fused is fused with the first target image by fusing corresponding pixels to obtain the fused image.

[0029] Optionally, the main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel.

[0030] Secondly, embodiments of this application provide an image fusion apparatus, comprising:

[0031] The acquisition module is used to acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel;

[0032] A correction module is used to perform distortion correction processing on the second target image using a target mapping relationship to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0033] The fusion module is used to perform image fusion on the first target image and the corrected second target image to obtain a fused image.

[0034] Optionally, the target mapping relationship is specifically a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to the target shape;

[0035] The target shape is the shape of the object represented by the image content of the first target image, as displayed in the first target image.

[0036] Optionally, the target mapping relationship is a coordinate mapping relationship generated through a predetermined calibration method;

[0037] The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object.

[0038] The two image sensors include the main channel image sensor and the secondary channel image sensor.

[0039] Optionally, the calibration process for the predetermined calibration method includes:

[0040] Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object;

[0041] The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship;

[0042] The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object.

[0043] Optionally, when the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape;

[0044] The actual shape of the calibration object is the shape obtained by projecting the target image shape using optical inverse projection.

[0045] Optionally, the method for determining the actual shape of the calibration object includes:

[0046] The shape parameters of the target image shape, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target image shape with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object.

[0047] The specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object.

[0048] Optionally, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel;

[0049] The fusion module is specifically used for:

[0050] The corrected second target image is used to identify a specified object region to obtain a candidate image corresponding to the specified object region.

[0051] Determine the image to be fused corresponding to the candidate images; wherein, the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate images;

[0052] The image to be fused is fused with the first target image by fusing corresponding pixels to obtain the fused image.

[0053] Optionally, the main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel.

[0054] Thirdly, embodiments of this application provide an electronic device, including:

[0055] Memory, used to store computer programs;

[0056] A processor, when executing a program stored in memory, implements any of the described image fusion methods.

[0057] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned image fusion methods.

[0058] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the image fusion methods described above.

[0059] Beneficial effects of the embodiments in this application:

[0060] The image fusion method provided in this application first acquires a first target image output from the main channel and a second target image output from the secondary channel. Distortion correction is then performed on the second target image using a target mapping relationship to obtain a distortion-corrected second target image. Finally, the first target image and the corrected second target image are fused to obtain the target image. Compared to related technologies that perform distortion correction on images from each channel before image fusion, this solution does not perform distortion correction on the first target image output from the main channel. Instead, distortion correction is performed on the second target image output from the secondary channel using a target mapping relationship. This ensures that the distortion level of the corrected second target image matches that of the first target image, thus enabling the fusion of the first target image and the distortion-corrected second target image without loss of the field of view of the first target image output from the main channel. Therefore, this solution can achieve image fusion without loss of the field of view of the image in a specified channel.

[0061] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0063] Figure 1 A flowchart illustrating an image fusion method provided in an embodiment of this application;

[0064] Figure 2 This is another schematic diagram of an image fusion method provided in an embodiment of this application;

[0065] Figure 3a This is a schematic diagram showing the shape of the marker in the second sample image provided in the embodiments of this application;

[0066] Figure 3b A schematic diagram of a specified shape provided for an embodiment of this application;

[0067] Figure 4 This is a schematic diagram of the structure of an image fusion device provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0070] Image fusion refers to the process of combining image data of the same object acquired by multiple image acquisition channels with image processing and computer technology to extract the most useful information from each channel and finally fuse them into a high-quality image. Multiple image acquisition channels include, but are not limited to, visible light image acquisition channels, thermal imaging channels, ultraviolet light image acquisition channels, and so on.

[0071] Before image fusion, the output image of each of the multiple image acquisition channels needs to be corrected for image distortion.

[0072] When performing distortion correction using related technologies, the image output from each channel needs to be corrected individually until the image is distortion-free. Furthermore, the field of view of the images from each image acquisition channel will be lost, making it impossible to achieve image fusion while ensuring that the field of view of the image from a specified channel is not lost.

[0073] Based on this, this application provides an image fusion method, apparatus, and electronic device that can achieve image fusion without loss of field of view of the image in a specified channel.

[0074] The image fusion method provided in this application will be introduced below.

[0075] The image fusion method provided in this application can be applied to electronic devices, which can be image acquisition devices or terminal devices, such as mobile phones or computers. This application does not limit the specific form of the electronic device. When the electronic device is an image acquisition device, it can have multiple image acquisition channels. The image acquisition device can perform image fusion on the images from multiple image acquisition channels and then output the fused image. That is, the image output by the image acquisition device to the user can be the fused image. When the electronic device is a terminal device, it can fuse images from different acquisition devices or multiple image acquisition channels of the same acquisition device to obtain a fused image. The image fusion method provided in this application can be applied to any scenario with image fusion requirements, such as fusing images of the same object acquired by the visible light channel and the thermal imaging channel.

[0076] Furthermore, the entity executing the image fusion method provided in this application can be an image fusion device. For example, the image fusion device can be functional software running on a terminal device or an image acquisition device, such as functional software for image fusion. When the image fusion device runs on a terminal device, it can fuse multi-channel images in the terminal device. These multi-channel images can be acquired from an image acquisition device or acquired by the terminal device itself; this is not limited thereto. When the image fusion device runs on an image acquisition device, it can fuse multi-channel images acquired by the image acquisition device.

[0077] This application provides an image fusion method that may include the following steps:

[0078] Acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel;

[0079] Using a target mapping relationship, distortion correction processing is performed on the second target image to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0080] The first target image and the corrected second target image are fused to obtain a fused image.

[0081] The image fusion method provided in this application first acquires a first target image output from the main channel and a second target image output from the secondary channel. Distortion correction is then performed on the second target image using a target mapping relationship to obtain a distortion-corrected second target image. Finally, the first target image and the corrected second target image are fused to obtain the target image. Compared to related technologies that perform distortion correction on images from each channel before image fusion, this solution does not perform distortion correction on the first target image output from the main channel. Instead, distortion correction is performed on the second target image output from the secondary channel using a target mapping relationship. This ensures that the distortion level of the corrected second target image matches that of the first target image, thus enabling the fusion of the first target image and the distortion-corrected second target image without loss of the field of view of the first target image output from the main channel. Therefore, this solution can achieve image fusion without loss of the field of view of the image in a specified channel.

[0082] The image fusion method provided in this application will be described exemplarily below with reference to the accompanying drawings.

[0083] like Figure 1 As shown, the image fusion method provided in this application may include the following steps:

[0084] S101: Acquire the first target image output by the main channel and the second target image output by the secondary channel;

[0085] The main channel is a pre-designated image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel.

[0086] The image fusion method provided in this application fuses images acquired by multiple image acquisition channels for the same object. In order to ensure that the field of view of the image of the specified channel is not lost, the image acquisition channel whose field of view of the image to be fused remains unchanged relative to the output image can be pre-specified as the main channel, and secondary channels in addition to the main channel can be used. The number of secondary channels can be one or more, which is not limited here. When there are multiple secondary channels, there are also multiple second target images, and the processing method for each second target image is the same.

[0087] For example, in one implementation, the main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel. Multiple image acquisition channels may include both infrared thermal imaging channels and visible light image acquisition channels. In this case, the main channel can be an infrared thermal imaging channel, and the secondary channels can be visible light image acquisition channels. These multiple image acquisition channels may also include ultraviolet image acquisition channels, which can also be used as secondary channels; however, this is not a limitation. In practical applications, one image acquisition channel can be selected as the main channel according to requirements, and the remaining image acquisition channels can be used as secondary channels; this is not limited here.

[0088] It is understandable that when performing image fusion, it is first necessary to acquire the first target image output by the main channel and the second target image output by the secondary channel. When this solution is applied to a terminal device, the first target image and the second target image can be acquired from the image acquisition device. When this solution is applied to an image acquisition device, the first target image output by the main channel and the second target image output by the secondary channel can be directly determined.

[0089] S102: Using the target mapping relationship, perform distortion correction processing on the second target image to obtain the corrected second target image;

[0090] Wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0091] It should be noted that the image acquired by any image acquisition channel will exhibit varying degrees of distortion relative to the acquired object. Furthermore, the degree of distortion will differ between images of the same object acquired by different image acquisition channels. To ensure that the first target image output by the main channel does not suffer from field-of-view loss, the distortion of the second target image output by the secondary channels can be corrected using the first target image as a basis. The distortion degree of the corrected second target image can then match that of the first target image. Therefore, a target mapping relationship can be used to correct the distortion of the second target image. Additionally, if there are multiple secondary channels, a target mapping relationship must be pre-generated for each channel to perform distortion correction on the second target image output by that secondary channel.

[0092] After obtaining the first target image and the second target image, the distortion correction process of the second target image can be performed using the pre-generated target mapping relationship to obtain the corrected second target image. Furthermore, since the target mapping relationship is a mapping relationship that corrects the distortion degree of the second target image to the same degree of distortion as the first target image, the distortion degree of the obtained corrected second target image is the same as that of the first target image. Subsequently, the first target image and the corrected second target image can be image fused.

[0093] For example, the target mapping relationship is specifically a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to the target shape;

[0094] The target shape is the shape of the object represented by the image content of the first target image, as displayed in the first target image.

[0095] It should be noted that the degree of image distortion can be represented by the shape of the object represented by the image content in the image. However, the shape of the object represented by the image content of the second target image before distortion correction is different from the target shape of the first target image. When performing distortion correction on the second target image, the target mapping relationship can be used to map each pixel corresponding to the object represented by the image content of the second target image. The shape of the image composed of the mapped pixels can be the same as the target shape of the first target image. In other words, the degree of distortion of the second target image after distortion correction matches that of the first target image.

[0096] For example, the object represented by the image content of the first target image can be displayed in the first target image in a rectangular shape, and the object represented by the image content of the second target image can be displayed in the second target image in other shapes. When performing distortion correction on the second target image using the target mapping relationship, the object represented by the image content of the second target image can be corrected to a rectangular shape in the second target image, thereby obtaining the distortion-corrected second target image.

[0097] Furthermore, this target mapping relationship can be pre-generated and recorded on the terminal device or image acquisition device, thereby enabling rapid acquisition of the target mapping relationship and facilitating distortion correction processing of the second target image. The specific method for generating the target mapping relationship will be detailed in subsequent examples and will not be elaborated upon here.

[0098] It should be noted that the above description of distortion correction processing of the second target image is merely an example and should not be construed as limiting this application.

[0099] S103: Perform image fusion on the first target image and the corrected second target image to obtain a fused image;

[0100] After distortion correction processing is performed on the second target image, the resulting corrected second target image has the same degree of distortion as the first target image. At this point, the first target image and the corrected second target image can be fused to obtain a fused image.

[0101] For example, in one implementation, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel;

[0102] Image fusion is performed on the first target image and the corrected second target image to obtain a fused image, including:

[0103] The corrected second target image is used to identify a specified object region to obtain a candidate image corresponding to the specified object region.

[0104] Determine the image to be fused corresponding to the candidate images; wherein, the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate images;

[0105] The image to be fused is fused with the first target image by fusing corresponding pixels to obtain the fused image.

[0106] It is understandable that image distortion correction will result in a loss of field of view. That is, after distortion correction of the second target image, the corrected second target image will lose some of its field of view. To ensure that the corrected second target image includes the region of the first target image, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel. When performing image fusion across multiple channels, this scheme can guarantee that the field of view of the main channel is not lost. Considering the tolerances of the image acquisition equipment and other factors, the field of view of the secondary channel can be set to be larger than that of the main channel; that is, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel. When performing distortion correction on the secondary channel, the field of view of the secondary channel will be lost. By setting the field of view of the secondary channel to be larger than that of the primary channel, it can be ensured that even if the field of view of the secondary channel is lost, the region in the secondary channel image corresponding to the output image of the primary channel will not be lost. Furthermore, during subsequent image fusion, the image of the secondary channel after distortion correction still has a margin that can be cropped. The image corresponding to the primary channel image can be selected from the image of the secondary channel after distortion correction and fused with the primary channel image. In other words, this solution can achieve image fusion without losing the field of view of the specified channel image.

[0107] During image fusion, a designated object region can be identified in the corrected second target image to obtain a candidate image. This designated object region can be the region represented by the first target image. Since the field of view of the secondary channel is larger than that of the primary channel, the size of the output second target image will also be larger than that of the first target image. The size of the candidate image may differ from that of the first target image. Therefore, the image to be fused corresponding to the candidate image can also be determined. Subsequently, the image to be fused can be fused with the corresponding pixels in the first target image to obtain the fused image. It should be noted that the designated object region can be the region corresponding to the target object contained in the first target image; for example, the target object can be a person or an object.

[0108] The field of view of any image output by the secondary channel is larger than that of any image output by the main channel. This allows the second target image output by the secondary channel to contain complete information representing the first target image. Furthermore, after distortion correction of the second target image, the corrected second target image still contains complete information representing the first target image; that is, the image corresponding to the region of the first target image in the second target image is not lost. By determining a fusion image of the same size as the first target image, the image corresponding to the region of the first target image can be selected from the corrected second target image, making it easier to fuse the first target image and the image to be fused.

[0109] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0110] It should be noted that the first target image and the second target image in this embodiment are from a publicly available dataset.

[0111] The image fusion method provided in this application first acquires a first target image output from the main channel and a second target image output from the secondary channel. Distortion correction is then performed on the second target image using a target mapping relationship to obtain a distortion-corrected second target image. Finally, the first target image and the corrected second target image are fused to obtain the target image. Compared to related technologies that perform distortion correction on images from each channel before image fusion, this solution does not perform distortion correction on the first target image output from the main channel. Instead, distortion correction is performed on the second target image output from the secondary channel using a target mapping relationship. This ensures that the distortion level of the corrected second target image matches that of the first target image, thus enabling the fusion of the first target image and the distortion-corrected second target image without loss of the field of view of the first target image output from the main channel. Therefore, this solution can achieve image fusion without loss of the field of view of the image in a specified channel.

[0112] Optionally, in another embodiment provided in this application, the target mapping relationship is a coordinate mapping relationship generated by a predetermined calibration method;

[0113] The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object.

[0114] The two image sensors include the main channel image sensor and the secondary channel image sensor.

[0115] It is understandable that the target mapping relationship can be generated through calibration. During calibration, the coordinate relationship between sample images obtained when the same calibration object is acquired by image sensors using the main channel and secondary channel can be used to calibrate the object.

[0116] Optionally, the calibration process for the predetermined calibration method includes:

[0117] Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object;

[0118] The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship;

[0119] The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object.

[0120] When generating a target mapping relationship through calibration, a second sample image obtained when the secondary channel acquires images of the calibration object can be acquired first. For this second sample image, the coordinate mapping relationship required to correct the display shape of the calibration object in the second sample image to a specified shape can be determined, thus obtaining the target mapping relationship. The specified shape can be the display shape of the calibration object in the first sample image obtained when the main channel sensor acquires images of the calibration object. In this case, the calibrated coordinate mapping relationship is the coordinate mapping relationship for correcting the display shape of the image content belonging to the calibration object in the secondary channel to the display shape of the image content belonging to the calibration object in the main channel. This coordinate mapping relationship can be used as the target mapping relationship for correcting the distortion degree of the secondary channel image to the distortion degree of the main channel image.

[0121] For example, such as Figure 3a As shown, the shape of the calibration object in the second sample image can be pincushion-shaped, such as... Figure 3b As shown, the specified shape can be a rectangle, which can be calibrated when generating target mapping relationships. Figure 3a The calibration of the pillow-shaped calibration object is corrected to Figure 3b The required coordinate mapping relationship when the rectangle shape is obtained is used as the target mapping relationship.

[0122] It is understood that there may be various ways to use coordinate mapping relationships when calibrating the display shape of the calibrator in the second sample image to a specified shape, and no limitation is made here.

[0123] For example, in one implementation, the display shape of the calibration object in the second sample image can be used as the image shape acquired during the camera calibration process. The pixel values ​​of each pixel corresponding to the specified shape can be used as the physical coordinate values ​​of the calibration object in the calibration process. The pixel values ​​of each pixel in the display shape of the calibration object in the second sample image can be identified. Based on the determined pixel values, by mapping each pixel in the display shape of the calibration object in the second sample image to each pixel corresponding to the specified shape, the coordinate mapping relationship required to correct the display shape of the calibration object in the second sample image to the specified shape can be determined, thereby obtaining the target mapping relationship.

[0124] Additionally, it's understandable that Zhang Zhengyou's calibration method can be used for camera calibration: The camera is used to capture images of the Zhang Zhengyou calibration grid from different angles, obtaining a set of images; feature points in the images are detected, and their pixel values ​​are obtained; based on the known size of the grid and the origin of the world coordinate system, the physical coordinates of the feature points are calculated; based on the relationship between the physical coordinates and pixel coordinates, the camera's intrinsic and extrinsic parameter matrices are solved; the camera's distortion parameters are then solved; and the Levenberg-Marquardt (LM) algorithm is used to optimize the camera's intrinsic and extrinsic parameter matrices, as well as the distortion parameters. Of course, other methods can also be used for camera calibration, which are not limited here.

[0125] It should be noted that the specified shape can be any regular shape, and any method that can determine the coordinate mapping relationship required to correct the display shape of the calibration object in the second sample image to the specified shape is applicable to this scheme and is not limited here. By calibrating the coordinate mapping relationship required to correct the display shape of the calibration object in the second sample image to the specified shape, the specified shape can be set to any regular shape, which simplifies the calibration of coordinate mapping relationships and enables the rapid generation of target mapping relationships, thereby improving the efficiency of subsequent image fusion.

[0126] Optionally, when the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape;

[0127] The actual shape of the calibration object is the shape obtained by projecting the target image shape using optical inverse projection.

[0128] It is understood that when the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained can be a pre-specified target graphic shape. This graphic shape can be the desired display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object, such as a rectangle, a rhombus, etc.

[0129] The shape of the calibration object can be obtained by projecting the target graphic shape using optical reverse projection. Of course, the actual shape of the calibration object can also be determined by other methods. Any method that can determine the actual shape of the calibration object is applicable to this scheme and is not limited here.

[0130] Optionally, the method for determining the actual shape of the calibration object includes:

[0131] The shape parameters of the target image shape, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target image shape with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object.

[0132] The specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object.

[0133] The actual shape of the calibration object can be obtained through optical software simulation using inverse projection. During imaging, the size and shape of the image plane can be determined by the object distance, the shape parameters of the object plane, and the optical parameters of the sensor in that channel. When determining the actual shape of the calibration object through inverse projection, the shape parameters of the target image, the specified optical parameters of the image sensor in the main channel, and the corresponding object distance information of the main channel can be determined first, thereby obtaining the actual shape of the calibration object. Furthermore, for example, the specified optical parameters can be the lens and sensor parameters of the main channel, such as: lens focal length, sensor pixel count, pixel size, distortion curve, etc.

[0134] Additionally, it should be noted that the calibration process for the predetermined calibration method also includes:

[0135] Acquire a first sample image and a second sample image; wherein the first sample image and the second sample image are respectively images obtained when the image sensors of the main channel and the secondary channel are used to acquire images of the calibration object; the actual shape of the calibration object is a pre-defined shape.

[0136] The target mapping relationship is obtained by calibrating the coordinate mapping relationship required to correct the display shape of the calibrator in the second sample image to the display shape of the calibrator in the first sample image.

[0137] When generating the target mapping relationship, the display shape of the calibrator in the second sample image and the display shape of the calibrator in the first sample image can be directly calibrated to obtain the target mapping relationship. At this time, the first sample image and the second sample image are the shapes obtained by the image acquisition of the calibrator in the main channel and the secondary channel in advance. It is not necessary to determine the actual shape of the calibrator through inverse projection, so the target mapping relationship can be generated quickly.

[0138] That is, the actual shape of the calibration object can be a pre-defined shape or a shape generated by optical reverse projection; there is no limitation here.

[0139] It should be noted that optical software simulation can accurately determine the actual shape of the calibration object, including parameters such as its size and shape. After obtaining the actual shape, the secondary channel can be used to acquire an image of the calibration object, resulting in a second sample image. The display shape of the calibration object in the second sample image can then be determined. Using this calibration method, the coordinate mapping relationship required to correct the display shape of the calibration object in the second sample image to a specified shape can be determined, thus obtaining the target mapping relationship. Compared to pre-setting the actual shape of the calibration object, optical software simulation is more effective in determining the target mapping relationship. When performing distortion correction on the second target image, the correction effect of the corrected second target image is better, and the image quality of the fused image is higher during image fusion.

[0140] The following description, in conjunction with another embodiment, provides an exemplary description of an image fusion method provided in this application.

[0141] like Figure 2 As shown, the image fusion method provided in this application may include the following steps:

[0142] S201: Determine the main channel and acquire the images of the main channel and the secondary channel; that is, determine the image acquisition channel that is pre-specified for image fusion and whose field of view remains unchanged relative to the output image, and acquire the images of the main channel and the secondary channel, corresponding to the acquisition of the first target image output by the main channel and the second target image output by the secondary channel as described above.

[0143] S202: Obtain the distortion correction mapping relationship; that is, obtain the mapping relationship for distortion correction of the sub-channel image, corresponding to the above-mentioned method of obtaining the pre-generated target mapping relationship. When obtaining the distortion correction mapping relationship, the second sample image of the sub-channel can be obtained first. By calibration, the display shape of the second sample image with respect to the calibrator is corrected to the coordinate mapping relationship required for the specified shape, thereby obtaining the distortion correction mapping relationship, corresponding to the above-mentioned method of determining the target mapping relationship.

[0144] S203: Perform distortion correction on the secondary channel image; that is, use the distortion correction mapping relationship obtained in S202 to perform distortion correction on the secondary channel image, and use the target mapping relationship mentioned above to perform distortion correction processing on the second target image to obtain the corrected second target image.

[0145] S204: The distortion-corrected secondary channel image is fused with the main channel image; that is, each pixel in the distortion-corrected secondary channel image is fused with each pixel in the corresponding main channel image, corresponding to the image fusion of the first target image and the corrected second target image described above, to obtain a fused image.

[0146] It should be noted that steps S201-S204 are similar to steps S101-S103 above, and can be referred to each other, so they will not be described in detail here.

[0147] This application provides an image fusion method that uses the main channel as a base, without distortion correction to ensure no loss in the field of view of the main channel. Using an acquired distortion correction mapping relationship, distortion correction is applied to the secondary channel images, ensuring that the distortion degree of the corrected secondary channel images is the same as that of the main channel images. Finally, image fusion is performed. This method achieves image fusion without loss of field of view in the main channel images.

[0148] Furthermore, for multi-channel image acquisition devices with a large field of view, the field of view requirements of the user's main channel can be prioritized, and the pain point of existing technologies requiring distortion correction for all channels during image fusion can be addressed. Any channel can be selected as the main channel based on requirements, ensuring the image quality of the main channel remains unchanged, and distortion correction is only applied to the secondary channel image. This matches the distortion level of the main channel image, reducing the number of distortion correction steps and minimizing errors caused by distortion correction.

[0149] Based on the above image fusion method, this application also provides an image fusion apparatus, such as... Figure 4 As shown, the device includes:

[0150] The acquisition module 410 is used to acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel;

[0151] The correction module 420 is used to perform distortion correction processing on the second target image using a target mapping relationship to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0152] The fusion module 430 is used to perform image fusion on the first target image and the corrected second target image to obtain a fused image.

[0153] The image fusion apparatus provided in this application first acquires a first target image output from the main channel and a second target image output from the secondary channel. It then uses a target mapping relationship to perform distortion correction on the second target image, obtaining a distortion-corrected second target image. Finally, it fuses the first target image and the corrected second target image to obtain the target image. Compared to related technologies that perform distortion correction on images from each channel before image fusion, this solution does not perform distortion correction on the first target image output from the main channel. Instead, it uses a target mapping relationship to correct the distortion of the second target image output from the secondary channel, ensuring that the distortion degree of the corrected second target image matches that of the first target image. This allows for the fusion of the first target image and the distortion-corrected second target image without loss of the field of view of the first target image output from the main channel. Therefore, this solution can achieve image fusion without loss of the field of view of the image in a specified channel.

[0154] Optionally, the target mapping relationship is specifically a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to the target shape;

[0155] The target shape is the shape of the object represented by the image content of the first target image, as displayed in the first target image.

[0156] Optionally, the target mapping relationship is a coordinate mapping relationship generated through a predetermined calibration method;

[0157] The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object.

[0158] The two image sensors include the main channel image sensor and the secondary channel image sensor.

[0159] Optionally, the calibration process for the predetermined calibration method includes:

[0160] Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object;

[0161] The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship;

[0162] The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object.

[0163] Optionally, when the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape;

[0164] The actual shape of the calibration object is the shape obtained by projecting the target image shape using optical inverse projection.

[0165] Optionally, the method for determining the actual shape of the calibration object includes:

[0166] The shape parameters of the target image shape, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target image shape with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object.

[0167] The specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object.

[0168] Optionally, the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel;

[0169] The fusion module is specifically used for:

[0170] The corrected second target image is used to identify a specified object region to obtain a candidate image corresponding to the specified object region.

[0171] Determine the image to be fused corresponding to the candidate images; wherein, the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate images;

[0172] The image to be fused is fused with the first target image by fusing corresponding pixels to obtain the fused image.

[0173] Optionally, the main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel.

[0174] This application also provides an electronic device, such as... Figure 5 As shown, it includes:

[0175] Memory 501 is used to store computer programs;

[0176] When processor 502 executes the program stored in memory 501, it performs the following steps:

[0177] Acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel whose field of view remains unchanged relative to the output image for image fusion, and the secondary channel is an image acquisition channel other than the main channel;

[0178] Using a target mapping relationship, distortion correction processing is performed on the second target image to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image;

[0179] The first target image and the corrected second target image are fused to obtain a fused image.

[0180] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 502, communication interface, and memory 501 communicating with each other via the communication bus.

[0181] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0182] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0183] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0184] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0185] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image fusion methods.

[0186] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image fusion methods described above.

[0187] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or other media (e.g., solid state disk (SSD)).

[0188] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0189] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0190] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that, include: Acquire a first target image output by the main channel and a second target image output by the secondary channel; wherein, the main channel is a pre-specified image acquisition channel used for image fusion, and the field of view of the image relative to the output image remains unchanged; the secondary channel is an image acquisition channel other than the main channel; the image acquired by each image acquisition channel is distorted relative to the acquired object, and the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel; Using a target mapping relationship, distortion correction processing is performed on the second target image to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image; the distortion degree of the second target image after distortion correction matches the distortion degree of the first target image; specifically, the target mapping relationship is a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to a target shape; the target shape is the display shape of the object represented by the image content of the first target image in the first target image; The corrected second target image is used to identify a specified object region to obtain a candidate image corresponding to the specified object region. Determine the image to be fused corresponding to the candidate images; wherein, the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate images; The image to be fused is fused with the first target image by fusing corresponding pixels to obtain the fused image.

2. The method according to claim 1, characterized in that, The target mapping relationship is a coordinate mapping relationship generated through a predetermined calibration method; The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object. The two image sensors include the main channel image sensor and the secondary channel image sensor.

3. The method according to claim 2, characterized in that, The calibration process for the predetermined calibration method includes: Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object; The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship; The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object.

4. The method according to claim 3, characterized in that, When the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape; The actual shape of the calibration object is the shape obtained by projecting the target graphic shape using optical reverse projection.

5. The method according to claim 4, characterized in that, The methods for determining the actual shape of the calibration object include: The shape parameters of the target graphic, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target graphic with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object. The specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object.

6. The method according to any one of claims 1-5, characterized in that, The main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel.

7. An image fusion apparatus, characterized in that, include: An acquisition module is used to acquire a first target image output by a main channel and a second target image output by a secondary channel; wherein, the main channel is a pre-specified image acquisition channel used for image fusion, and the field of view of the image relative to the output image remains unchanged; the secondary channel is an image acquisition channel other than the main channel; the image acquired by each image acquisition channel is distorted relative to the acquired object, and the field of view of any image output by the main channel is smaller than the field of view of any image output by the secondary channel; A correction module is used to perform distortion correction processing on the second target image using a target mapping relationship to obtain a corrected second target image; wherein, the target mapping relationship is a mapping relationship used to correct the distortion degree of the second target image to the distortion degree of the first target image; the distortion degree of the second target image after distortion correction matches the distortion degree of the first target image; specifically, the target mapping relationship is a mapping relationship used to correct the display shape of the object represented by the image content of the second target image in the second target image to a target shape; the target shape is the display shape of the object represented by the image content of the first target image in the first target image; The fusion module is used to identify a specified object region in the corrected second target image to obtain a candidate image corresponding to the specified object region; determine the image to be fused corresponding to the candidate image, wherein the image to be fused is an image with the same size as the first target image obtained by adjusting the size of the candidate image; and fuse the image to be fused with the first target image by fusing corresponding pixels to obtain a fused image.

8. The apparatus according to claim 7, characterized in that, The target mapping relationship is a coordinate mapping relationship generated through a predetermined calibration method; The predetermined calibration method is a calibration method based on the coordinate relationship between sample images obtained when two image sensors are used to acquire images of the same calibration object, with respect to the calibration object. The two image sensors include the main channel image sensor and the secondary channel image sensor; And / or, The calibration process for the predetermined calibration method includes: Acquire a second sample image; wherein the second sample image is: the image obtained when the image sensor of the secondary channel acquires an image of the calibration object; The coordinate mapping relationship required to correct the display shape of the calibrated object in the second sample image to a specified shape is used to obtain the target mapping relationship; The specified shape is: the display shape of the calibration object in the first sample image obtained when the image sensor of the main channel is used to acquire an image of the calibration object; And / or, When the image sensor of the main channel is used to acquire an image of the calibration object, the display shape of the calibration object in the first sample image obtained is a pre-specified target graphic shape; The actual shape of the calibration object is the shape obtained by projecting the target graphic shape using optical reverse projection. And / or, The methods for determining the actual shape of the calibration object include: The shape parameters of the target graphic, the specified optical parameters of the image sensor of the main channel, and the object distance information corresponding to the main channel are input into the optical simulation software so that the optical simulation software performs optical back projection on the target graphic with the shape parameters based on the object distance information corresponding to the main channel and the specified optical parameters to obtain the actual shape of the calibration object. Wherein, the specified optical parameter is an optical parameter that is related to the generation of image distortion when the image content in any image output by the main channel produces image distortion relative to the corresponding acquisition object; And / or, The main channel is an infrared thermal imaging channel, and the secondary channel is a visible light image acquisition channel.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the image fusion method according to any one of claims 1-6.

Images