An image fusion method, apparatus and device

Abstract

The application provides an image fusion method, device and equipment. The image fusion method provided by the application comprises: performing distortion correction on an obtained initial visible light image and an initial thermal imaging image to obtain a corrected visible light image and a corrected thermal imaging image; performing partition processing on the corrected visible light image and the corrected thermal imaging image synchronously; for each partition, determining a corresponding feature point pair of the partition according to a sub-region of the corrected visible light image corresponding to the partition and a sub-region of the corrected thermal imaging image corresponding to the partition; determining a corresponding transformation matrix of the partition according to the corresponding feature point pair of the partition, and performing registration on the sub-regions of the corrected visible light image corresponding to the partitions according to the corresponding transformation matrices of the partitions to obtain a registered visible light image; and fusing the registered visible light image and the corrected thermal imaging image to obtain a fused image. The image fusion method, device and equipment provided by the application can improve the overall consistency of the fused image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method, apparatus and device. Background Technology

[0002] Thermal infrared images are obtained by collecting and recording the infrared radiation emitted by an object using a thermal infrared imager, and then converting the images. Due to the positive correlation between infrared radiation and object temperature, the color or grayscale values ​​of thermal infrared images can provide an intuitive understanding of the temperature distribution of the scene being photographed. However, because thermal infrared images have low contrast and strong spatial correlation, they are not good at reflecting the detailed features of a target.

[0003] In recent years, visible light images and thermal infrared images have often been fused to leverage the advantages of each type of image and compensate for the shortcomings of single imaging. However, traditional image fusion methods often result in poor consistency and detail matching in the fused image. Summary of the Invention

[0004] In view of this, this application provides an image fusion method, apparatus, and device to improve the overall consistency and detail matching of fused images.

[0005] A first aspect of this application provides an image fusion method, the image fusion method comprising:

[0006] An initial visible light image and an initial thermal image of the same scene are acquired, and distortion correction is performed on the initial visible light image and the initial thermal image respectively to obtain a corrected visible light image and a corrected thermal image; the corrected visible light image and the corrected thermal image have the same resolution;

[0007] The corrected visible light image and the corrected thermal imaging image are simultaneously processed by partitioning.

[0008] For each partition, based on the sub-region of the partition corresponding to the corrected visible light image and the sub-region of the partition corresponding to the corrected thermal imaging image, the feature point pair corresponding to the partition is determined;

[0009] The transformation matrix corresponding to the partition is determined based on the feature point pairs corresponding to the partition, and the sub-regions of the corrected visible light image corresponding to each partition are registered based on the transformation matrix corresponding to each partition to obtain the registered visible light image.

[0010] The registered visible light image and the corrected thermal imaging image are fused to obtain a fused image.

[0011] A second aspect of this application provides an image fusion apparatus, which includes an acquisition module, a partitioning module, a processing module, and a fusion module; wherein...

[0012] The acquisition module is used to acquire an initial visible light image and an initial thermal imaging image of the same scene captured by a binocular camera, and to perform distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image; the corrected visible light image and the corrected thermal imaging image have the same resolution;

[0013] The partitioning module is used to simultaneously perform partitioning processing on the corrected visible light image and the corrected thermal imaging image;

[0014] The processing module is used to determine the feature point pair corresponding to each partition based on the sub-region of the corrected visible light image corresponding to the partition and the sub-region of the corrected thermal imaging image corresponding to the partition.

[0015] The processing module is further configured to determine the transformation matrix corresponding to the partition based on the feature point pairs corresponding to the partition, and to register the sub-regions of the corrected visible light image corresponding to each partition based on the transformation matrix corresponding to each partition, so as to obtain the registered visible light image.

[0016] The fusion module is used to fuse the registered visible light image and the corrected thermal imaging image to obtain a fused image.

[0017] A third aspect of this application provides an image fusion apparatus, the image fusion apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the image fusion methods provided in the first aspect of this application.

[0018] The image fusion method, apparatus, and device provided in this application acquire an initial visible light image and an initial thermal imaging image of the same scene, and respectively perform distortion correction on the initial visible light image and the initial thermal imaging image to obtain a corrected visible light image and a corrected thermal imaging image. Then, the corrected visible light image and the corrected thermal imaging image are simultaneously partitioned. For each partition, feature point pairs corresponding to that partition are determined based on the sub-regions of the corrected visible light image and the corrected thermal imaging image corresponding to that partition. A transformation matrix corresponding to that partition is determined based on the feature point pairs. The sub-regions of the corrected visible light image corresponding to each partition are registered according to the transformation matrices of each partition to obtain a registered visible light image. Finally, the registered visible light image and the corrected thermal imaging image are fused to obtain a fused image. In this way, by first correcting the distortion of the initial visible light image and the initial thermal imaging image, the curved edges caused by lens distortion can be restored to straight lines, eliminating the interference of distortion on subsequent registration and providing a foundation for high-precision registration. Furthermore, by constructing a sufficient number of feature point pairs one by one through partitioning and solving the corresponding transformation matrix, accurate registration by region can be achieved. This effectively reduces the positional deviation between visible light images and thermal imaging images, strengthens the correspondence between local regions of the two images, and significantly improves the spatial consistency and detail matching accuracy between images. Based on this, image fusion can avoid problems such as local misalignment and detail misalignment, ultimately resulting in a fused image with accurate registration, natural detail connection, and better overall consistency.

[0019] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 A flowchart of an embodiment of the image fusion method provided in this application;

[0022] Figure 2 This is a schematic diagram illustrating the implementation principle of partitioning processing in an exemplary embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating feature point pairs corresponding to each partition in an exemplary embodiment of this application;

[0024] Figure 4 A flowchart of Embodiment 2 of the image fusion method provided in this application;

[0025] Figure 5 A flowchart of Embodiment 3 of the image fusion method provided in this application;

[0026] Figure 6 A flowchart of Embodiment 4 of the image fusion method provided in this application;

[0027] Figure 7 This is a hardware structure diagram of an image fusion device, which is the embodiment of an exemplary embodiment of this application.

[0028] Figure 8 This is a schematic diagram of an image fusion apparatus shown as an exemplary embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0030] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0032] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0033] Figure 1 This is a flowchart of an embodiment of the image fusion method provided in this application. Please refer to... Figure 1 The image fusion method provided in this embodiment includes:

[0034] S101. Acquire an initial visible light image and an initial thermal imaging image in the same scene, and perform distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image; the corrected visible light image and the corrected thermal imaging image have the same resolution.

[0035] Specifically, the image fusion method and apparatus provided in this embodiment are applied to an image fusion device. The image fusion device can be deployed on a stereo camera or on other devices independent of the stereo camera, such as a server. In this embodiment, it is not limited to this.

[0036] Referring to the preceding description, when the image fusion device is deployed on a stereo camera, it directly acquires the initial visible light image and initial thermal image of the same scene captured by the stereo camera at a certain moment. Furthermore, when the image fusion device is deployed on other devices, those other devices can acquire the initial visible light image and initial thermal image of the same scene captured by the stereo camera at a certain moment from the stereo camera.

[0037] Furthermore, after acquiring the initial visible light image and the initial thermal imaging image, distortion correction is performed on the initial visible light image and the initial thermal imaging image respectively to obtain the corrected visible light image and the corrected thermal imaging image. Optionally, in one possible implementation, the process of performing distortion correction on the initial visible light image and the initial thermal imaging image respectively includes:

[0038] (1) The initial visible light image is distorted by using a first set of distortion coefficients corresponding to a pre-determined visible light image to obtain a distorted visible light image.

[0039] (2) The initial thermal infrared image is distorted by using a pre-determined set of second distortion coefficients corresponding to the thermal infrared image to obtain a distorted thermal infrared image; at least one distortion coefficient in the second set of distortion coefficients is different from that in the first set of distortion coefficients.

[0040] It is understandable that the first set of distortion coefficients corresponding to the pre-determined visible light image and the second set of distortion coefficients corresponding to the pre-determined thermal infrared image are determined according to the following method: For a pre-set checkerboard pattern, by taking checkerboard images from different angles, the image coordinates and world coordinates of the grid points are extracted (the world coordinates are the actual physical coordinates of the checkerboard pattern), and the distortion parameter set of the binocular camera (the distortion parameter set includes radial distortion coefficients and tangential distortion coefficients) can be solved.

[0041] It should be noted that when determining the first distortion coefficient corresponding to the visible light image, the pre-set checkerboard grid is a regular black and white checkerboard grid; while when determining the second distortion coefficient corresponding to the thermal infrared image, the pre-set checkerboard grid is an infrared checkerboard grid.

[0042] It is understood that the first set of distortion coefficients is used to characterize the distortion characteristics of the visible light imaging system, and the second set of distortion coefficients is used to characterize the distortion characteristics of the thermal infrared imaging system. Since the lens structure, focal length parameters and operating bands of the two imaging systems are different, at least one distortion coefficient in the first set of distortion coefficients and the second set of distortion coefficients are different.

[0043] In practice, the first set of distortion coefficients can be used to perform reverse distortion mapping on each pixel in the initial visible light image. The corresponding original pixel position can be calculated based on the radial distortion parameters and tangential distortion parameters in the first set of distortion coefficients, and the corrected pixel value can be obtained through interpolation reconstruction, thereby generating a corrected visible light image.

[0044] Similarly, using the second set of distortion coefficients, distortion correction processing is performed on the initial thermal infrared image to generate a corrected thermal infrared image.

[0045] It should be noted that the corrected visible light image and the corrected thermal imaging image have the same resolution. In practice, the resolution of the two images can be unified before or after distortion correction; this embodiment does not limit this.

[0046] S102. The corrected visible light image and the corrected thermal imaging image are simultaneously processed by partitioning.

[0047] Optionally, in one possible implementation, the calibrated visible light image and the calibrated thermal imaging image are simultaneously partitioned according to a preset partitioning rule; wherein the preset partitioning rule is used to indicate that the image is divided into multiple partitions.

[0048] It should be noted that the preset partitioning rules are set according to actual needs, and their specific content is not limited in this embodiment. For example, in one possible implementation, the preset partitioning rules are used to indicate that the image is divided into a first partition and a second partition; wherein, the first partition is the outer region; and the second partition is the inner region surrounded by the outer region.

[0049] Figure 2 This is a schematic diagram illustrating the implementation principle of partitioning processing in an exemplary embodiment of this application. Please refer to... Figure 2 ,exist Figure 2In the example shown, the image (which includes a calibrated visible light image and a calibrated thermal infrared image) is divided into a first partition and a second partition. The first partition is the outer region, and the second partition is the inner region.

[0050] It should be noted that the size of the outer area is set according to actual needs.

[0051] For example, in another possible implementation, the preset partitioning rule is used to instruct the image to be partitioned along a first center line and / or a second center line; wherein the first center line is a center line in the width direction; and the second center line is a center line in the length direction.

[0052] S103. For each partition, determine the feature point pair corresponding to the partition based on the sub-region of the corrected visible light image corresponding to the partition and the sub-region of the corrected thermal imaging image corresponding to the partition.

[0053] Specifically, the number of feature point pairs corresponding to each partition is greater than a preset threshold. The preset threshold is greater than or equal to 4. Optionally, in one possible implementation, the preset threshold is 4.

[0054] Optionally, in one possible implementation, the calibrated visible light image and the calibrated thermal imaging image can be displayed to the user; then, in response to the user's annotation operation, the feature point pair consisting of the visible light feature points marked by the user on the sub-region of the calibrated visible light image corresponding to the partition, and the thermal infrared feature points corresponding to the visible light feature points marked by the user on the sub-region of the calibrated thermal imaging image corresponding to the partition, is determined as the feature point pair corresponding to the partition.

[0055] For specific implementation details, please refer to [link / reference]. Figure 2 To improve the accuracy of the correspondence between feature point pairs, manual annotation can be used to obtain them. Specifically, the calibrated visible light image and the calibrated thermal imaging image are first presented to the user in a side-by-side or top-bottom display manner, with each partition displayed synchronously. Then, the user can calibrate visible light feature points in the sub-region of the calibrated visible light image corresponding to that partition. Simultaneously, the user will calibrate the corresponding thermal infrared feature point in the sub-region of the calibrated thermal infrared image corresponding to that partition. Further, in response to the user's calibration operation, the coordinates of the user-selected visible light feature point and the coordinates of the corresponding thermal infrared feature point are obtained, thus generating a feature point pair.

[0056] Specifically, during calibration, users can annotate locations such as building corners, equipment edges, intersections of target contours, road boundary points, or other salient locations that can be identified in both modal images. The image fusion device will combine the coordinates of visible light feature points with the coordinates of the corresponding thermal infrared feature points to form a set of feature point pairs, and determine multiple feature point pairs as the feature point pair set corresponding to that partition.

[0057] It should be noted that although the visible light image and the thermal imaging image originate from the same scene, due to different imaging mechanisms, the grayscale distribution, texture features, and edge features of the same target often differ significantly between the two images. In this embodiment, by manually calibrating and leveraging the user's ability to recognize scene correspondences, an accurate spatial correspondence between the visible light image and the thermal imaging image can be directly established, thereby obtaining high-precision feature point pairs. This allows for the subsequent calculation of the transformation matrix corresponding to the partition using these high-precision feature point pairs, effectively improving local registration accuracy, reducing registration errors caused by modal differences, and ultimately enhancing the fusion effect of the final fused image.

[0058] In addition, after the user completes the annotation, the consistency of the annotated feature point pairs can be checked, and outliers that deviate significantly from the spatial geometric relationship can be removed. Then, the remaining feature point pairs can be used to solve the transformation matrix corresponding to the partition. In this way, the accuracy of manual annotation can be fully utilized, and the robustness of the subsequent image registration process can be guaranteed.

[0059] S104. Determine the transformation matrix corresponding to the partition based on the feature point pairs corresponding to the partition, and register the sub-regions of the corrected visible light image corresponding to each partition based on the transformation matrix corresponding to each partition to obtain the registered visible light image.

[0060] Figure 3 This is a schematic diagram illustrating feature point pairs corresponding to each partition in an exemplary embodiment of this application. Please refer to... Figure 3 In one embodiment, the image (which includes a calibrated visible light image and a calibrated thermal infrared image) is divided into a first partition and a second partition. The first partition is the outer region, and the second partition is the inner region.

[0061] Furthermore, four pairs of feature points are set in both the first and second partitions. Please refer to [reference needed]. Figure 3 In one possible implementation, for the outer region, four pairs of feature points can be identified at the four corners of the image; further, for the inner region, four pairs of feature points can be identified at the four corners of the inner region. This embodiment does not limit these implementations.

[0062] After determining the feature point pairs corresponding to each partition, a local transformation relationship can be established for each partition. Specifically, for any partition, using the feature point pairs formed by the visible light image feature points and the thermal infrared image feature points within that partition, the local transformation relationship corresponding to that partition is solved through least squares fitting and other methods to obtain the transformation matrix corresponding to that partition. The transformation matrix is ​​used to describe the spatial mapping relationship between the visible light image and the thermal infrared image within that partition. The specific implementation principle and process of solving the transformation matrix can be found in the description in related technologies, and will not be repeated here.

[0063] Subsequently, based on the transformation matrix corresponding to each partition, the sub-regions of the corresponding partitions in the corrected visible light image are registered to obtain the sub-registered images corresponding to each partition. Then, the sub-registered images corresponding to each partition are stitched together to obtain the registered visible light image.

[0064] It should be noted that, due to potential differences in the transformation matrices corresponding to adjacent partitions, pixel discontinuities, abrupt brightness changes, or texture misalignments may occur at the partition boundaries when stitching the sub-registered images corresponding to each partition. Therefore, in one possible implementation, after stitching the sub-registered images corresponding to each partition, a smooth transition processing can be performed on the boundary regions of adjacent partitions. Specifically, a transition region can be constructed within the boundary region, and weighted fusion, linear interpolation, or other image smoothing algorithms can be used to fuse the pixel values ​​within the transition region to reduce visual abrupt changes caused by partition boundaries and improve the continuity and integrity of the registered images. The registered visible light image obtained after smoothing can maintain local registration accuracy while avoiding the impact of partition stitching marks on subsequent image fusion.

[0065] As described above, it is understood that in this embodiment, since the transformation matrix is ​​solved independently for each partition, the regional registration error caused by lens distortion, parallax variation, depth of field difference, and local nonlinear deformation can be effectively compensated. Finally, the registered sub-regions of each partition are stitched together according to the original spatial layout to obtain a registered visible light image that precisely corresponds to the calibrated thermal infrared image, thereby improving the accuracy of subsequent image fusion.

[0066] S105. The registered visible light image and the corrected thermal imaging image are fused to obtain a fused image.

[0067] Since the registered visible light image and the corrected thermal imaging image have been spatially aligned, corresponding pixel positions in the two images can represent the same scene region. Based on this, pixel-level information fusion can be performed on the two images. In specific implementation, the two images are fused according to a preset fusion rule, so that the fusion result simultaneously retains the rich scene details of the visible light image and the prominent thermal radiation features of the thermal imaging image.

[0068] Through the above fusion process, the obtained fused image simultaneously contains the texture, color, and structural information of the visible light image and the temperature distribution information of the thermal imaging image, thereby improving the accuracy and reliability of target detection, target recognition, and scene perception in complex environments.

[0069] The image fusion method provided in this embodiment acquires an initial visible light image and an initial thermal imaging image of the same scene, and performs distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image. Then, the corrected visible light image and the corrected thermal imaging image are simultaneously partitioned. For each partition, feature point pairs corresponding to the sub-regions of the corrected visible light image and the corrected thermal imaging image are determined. A transformation matrix corresponding to the partition is determined based on the feature point pairs. The sub-regions of the corrected visible light image corresponding to each partition are registered according to the transformation matrices of each partition to obtain a registered visible light image. Finally, the registered visible light image and the corrected thermal imaging image are fused to obtain a fused image. In this way, firstly, distortion correction of the initial visible light image and the initial thermal imaging image can restore curved edges caused by lens distortion to straight lines, eliminating the interference of distortion on subsequent registration and providing a foundation for high-precision registration. Furthermore, by constructing a sufficient number of feature point pairs one by one through partitioning and solving the corresponding transformation matrix, accurate registration by region can be achieved. This effectively reduces the positional deviation between visible light images and thermal imaging images, strengthens the correspondence between local regions of the two images, and significantly improves the spatial consistency and detail matching accuracy between images. Based on this, image fusion can avoid problems such as local misalignment and detail misalignment, ultimately resulting in a fused image with accurate registration, natural detail connection, and better overall consistency.

[0070] Figure 4 This is a flowchart of Embodiment 2 of the image fusion method provided in this application. Please refer to... Figure 4 The method provided in this embodiment, based on the above embodiments, includes the following step: simultaneously performing partitioning processing on the corrected visible light image and the corrected thermal imaging image, comprising:

[0071] S401. Perform foreground segmentation processing on the corrected visible light image to obtain foreground segmentation results, and determine the first partitioning rule based on the foreground segmentation results.

[0072] In practice, relying on the rich texture, color, and grayscale gradient information of the corrected visible light image, the foreground target region and invalid background region in the corrected visible light image are accurately distinguished through threshold segmentation, edge detection, or semantic segmentation algorithms. The foreground region is extracted and the pixel-level annotated foreground segmentation result is output.

[0073] For example, in one possible implementation, pixel-level foreground binary segmentation is performed on the corrected visible light image to obtain a standardized binary segmentation mask. Further, foreground targets are extracted based on the binary segmentation mask, and complete foreground parameter information is accurately statistically analyzed and output. This information mainly includes the total number of connected components, the minimum bounding box corresponding to each connected component, the area of ​​a single connected component region, and the coordinates of the target center, providing accurate data support for subsequent partitioning rule generation.

[0074] Furthermore, based on the region boundaries and location of each connected component, a first partitioning rule is determined. For example, the number of connected components in the segmentation results is counted, and for each connected component, a target partition that fits the target region is generated based on its contour boundary; the remaining background region of the image is treated as a whole as a background partition; the coordinate range, boundary size, and partition number of all target partitions and background partitions are integrated to form a partitioning rule. This partitioning rule is used to indicate that a connected component is treated as a partition, and the remaining background region is treated as a partition.

[0075] S402. The corrected visible light image and the thermal imaging image are synchronously partitioned according to the first partitioning rule.

[0076] In this step, after obtaining the first partitioning rule, the calibrated visible light image and thermal imaging image are simultaneously partitioned according to the first partitioning rule.

[0077] The image fusion method provided in this embodiment first adaptively determines the first partitioning rule based on the foreground segmentation result of the calibrated visible light image, and then performs partitioning processing on the calibrated visible light image and the calibrated thermal imaging image simultaneously based on the first partitioning rule. This enables the same source target and corresponding background region of the visible light image and the thermal imaging image to achieve a complete one-to-one match of the partitioning boundary, complete pixel-level local accurate registration, effectively correct the slight offset between the two modal images, and effectively avoid the problems of traditional registration methods being unable to adapt to complex dynamic scenes and local registration failure.

[0078] Figure 5 The flowchart for Embodiment 3 of the image fusion method provided in this application is shown below. Please refer to... Figure 5 The image fusion method provided in this embodiment, based on the above embodiments, includes the following step: simultaneously performing partitioning processing on the corrected visible light image and the corrected thermal imaging image, including:

[0079] S501. Identify the temperature level to which each pixel in the corrected thermal imaging image belongs based on the range of thermal radiation quantization values ​​corresponding to each of the preset multiple temperature levels.

[0080] It should be noted that the preset multiple temperature levels are set according to actual needs, and are not limited in this embodiment. For example, in one possible implementation, the preset multiple temperature levels include high, medium, and low levels. Furthermore, the range of thermal radiation quantization values ​​corresponding to each temperature level in the preset multiple temperature levels is also set according to actual needs, and is not limited in this embodiment. It is understood that the ranges of thermal radiation quantization values ​​corresponding to each temperature level in the preset multiple temperature levels are continuous and non-overlapping, and the intersection of the ranges of thermal radiation quantization values ​​corresponding to each temperature level in the preset multiple temperature levels covers all possible values ​​of the thermal radiation quantization value.

[0081] In this step, it is determined which temperature level corresponds to which thermal radiation quantization range the thermal radiation quantization value of each pixel in the corrected thermal imaging image belongs to, and then that temperature level is determined as the temperature level to which the pixel belongs.

[0082] S502. Determine the second partitioning rule based on the temperature level of each pixel in the corrected thermal imaging image.

[0083] After determining the temperature level of each pixel, pixels at the same temperature level are grouped into the same partition. In other words, the second partitioning rule is used to indicate that pixels at the same temperature level should be grouped into the same partition. Referring to the previous example, when there are multiple preset temperature levels including high, medium, and low, in this embodiment, high-level pixels are grouped into the same partition, medium-level pixels are grouped into the same partition, and low-level pixels are grouped into the same partition.

[0084] S503. The corrected visible light image and the thermal imaging image are synchronously partitioned according to the second partitioning rule.

[0085] Based on the description in step S502, after determining the second partitioning rule, this step involves synchronously partitioning the corrected visible light image and the thermal imaging image according to the second partitioning rule.

[0086] The image fusion method provided in this embodiment identifies the temperature level of each pixel in the calibrated thermal imaging image based on the thermal radiation quantization value range corresponding to each of a preset multiple temperature levels. Then, it determines a second partitioning rule based on the temperature level of each pixel in the calibrated thermal imaging image, and simultaneously partitions the calibrated visible light image and the thermal imaging image according to the second partitioning rule. This allows for accurate and adaptive partitioning by combining temperature features, avoiding the problem of fixed partitioning methods not matching actual scene features. Furthermore, by relying on the temperature attributes of pixels from the same source to unify the partitioning logic, it ensures that the partition boundaries and area ranges of the visible light image and the thermal imaging image are completely aligned, effectively improving the consistency of subsequent image fusion.

[0087] Figure 6 For the flowchart of Embodiment 4 of the image fusion method provided in this application, please refer to... Figure 6 The image fusion method provided in this embodiment, based on the above embodiments, includes determining the feature point pair corresponding to the partition based on the sub-region of the corrected visible light image corresponding to the partition and the sub-region of the corrected thermal imaging image corresponding to the partition, comprising:

[0088] S601. Feature point identification is performed on the sub-regions of the corrected visible light image corresponding to the partition to obtain a first set of feature points.

[0089] Specifically, image feature detection algorithms can be used to extract unique and stable pixels such as edges, corners, and texture inflection points within each sub-region. These pixels are not easily affected by imaging differences and can be used as reference markers for image registration.

[0090] For example, in one possible implementation, the SIFT algorithm can be used for feature point recognition.

[0091] S602. Feature point identification is performed on the sub-regions of the corrected thermal imaging image corresponding to the partition to obtain a second set of feature points.

[0092] In specific implementation, as described in step S601, an image feature detection algorithm can be used to extract feature points from each sub-region. It is understood that thermal infrared images rely on temperature differences to form grayscale distributions, and the image feature detection algorithm mainly captures features such as edges and contour inflection points formed by sudden temperature changes, serving as a registration reference for thermal imaging.

[0093] Referring to the previous example, the SIFT algorithm can also be used for feature point recognition in this step.

[0094] S603. Perform feature point matching on the first feature point set and the second feature point set to obtain a matching feature point pair.

[0095] Specifically, a feature descriptor matching algorithm can be used to calculate the similarity of each feature point in the first and second feature point sets, thus associating and pairing feature points in the two images that correspond one-to-one in terms of spatial location and morphological features, and establishing a pixel-level correspondence. For example, feature descriptors of the two sets of feature points can be extracted one by one to complete the similarity calculation and filtering, and finally generate one-to-one matching feature point pairs.

[0096] S604. When the number of matching feature point pairs is greater than or equal to a preset threshold, the matching feature point pair is determined as the feature point pair corresponding to the partition.

[0097] Referring to the preceding description, the preset threshold is set according to actual needs, and is not limited in this embodiment. For example, in one embodiment, the preset threshold is greater than or equal to 4.

[0098] S605. When the number of matched feature point pairs is less than a preset threshold, a reminder message is output to the user to remind the user to calibrate the feature point pairs.

[0099] Specifically, when the number of identified matching feature point pairs is less than a preset threshold, manual intervention is required, and the user will label the feature point pairs.

[0100] S606. The matching feature point pair and the user-defined feature point pair are determined as the feature point pair corresponding to the partition.

[0101] In this step, the identified matching feature point pairs and the user-defined feature point pairs are determined as the feature point pairs corresponding to the partition.

[0102] The method provided in this embodiment combines automatic matching with manual calibration, which not only gives full play to the efficiency of automatic processing, but also makes up for the shortcomings of poor automatic matching in complex areas. It ensures that each partition has a sufficient number of reliable feature point pairs, laying the foundation for subsequent solution of transformation matrix and accurate image registration, and can effectively improve the overall registration accuracy and image fusion quality.

[0103] Corresponding to the aforementioned embodiment of the image fusion method, this application also provides an embodiment of an image fusion apparatus.

[0104] An embodiment of the image fusion apparatus disclosed in this application can be applied to an image fusion device. The apparatus embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the image fusion device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 As shown, Figure 7This is a hardware structure diagram of an image fusion apparatus, which is part of an exemplary embodiment of this application. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the image fusion device in the embodiment may also include other hardware depending on the actual function of the image fusion device, which will not be described in detail here.

[0105] Figure 8 This is a schematic diagram of the structure of an embodiment of the image fusion apparatus provided in this application. Please refer to... Figure 8 The apparatus provided in this embodiment includes an acquisition module 810, a partitioning module 820, a processing module 830, and a fusion module 840; wherein,

[0106] The acquisition module 810 is used to acquire an initial visible light image and an initial thermal imaging image of the same scene captured by a binocular camera, and to perform distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image; the corrected visible light image and the corrected thermal imaging image have the same resolution;

[0107] The partitioning module 820 is used to simultaneously perform partitioning processing on the corrected visible light image and the corrected thermal imaging image;

[0108] The processing module 830 is used to determine, for each partition, a pair of feature points corresponding to that partition based on the sub-region of the corrected visible light image corresponding to that partition and the sub-region of the corrected thermal imaging image corresponding to that partition; the number of feature point pairs corresponding to each partition is greater than a preset threshold.

[0109] The processing module 830 is further configured to determine the transformation matrix corresponding to the partition based on the feature point pairs corresponding to the partition, and to register the sub-regions of the corrected visible light image corresponding to each partition based on the transformation matrix corresponding to each partition, so as to obtain the registered visible light image.

[0110] The fusion module 840 is used to fuse the registered visible light image and the corrected thermal imaging image to obtain a fused image.

[0111] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0112] Please continue to refer to Figure 7 This application also provides an image fusion apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0113] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0114] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that, The image fusion method includes: An initial visible light image and an initial thermal image of the same scene are acquired, and distortion correction is performed on the initial visible light image and the initial thermal image respectively to obtain a corrected visible light image and a corrected thermal image; the corrected visible light image and the corrected thermal image have the same resolution; The corrected visible light image and the corrected thermal imaging image are simultaneously processed by partitioning. For each partition, based on the sub-region of the partition corresponding to the corrected visible light image and the sub-region of the partition corresponding to the corrected thermal imaging image, the feature point pair corresponding to the partition is determined; The transformation matrix corresponding to the partition is determined based on the feature point pairs corresponding to the partition, and the sub-regions of the corrected visible light image corresponding to each partition are registered based on the transformation matrix corresponding to each partition to obtain the registered visible light image. The registered visible light image and the corrected thermal imaging image are fused to obtain a fused image.

2. The method according to claim 1, characterized in that, The simultaneous partitioning of the corrected visible light image and the corrected thermal imaging image includes... The calibrated visible light image and the calibrated thermal imaging image are simultaneously partitioned according to a preset partitioning rule; wherein, the preset partitioning rule is used to indicate that the image is divided into multiple partitions.

3. The method according to claim 2, characterized in that, The preset partitioning rule is used to indicate that the image is divided into a first partition and a second partition; wherein, the first partition is the outer region; and the second partition is the inner region surrounded by the outer region. or, The preset partitioning rule is used to instruct the image to be partitioned along the first center line and / or the second center line of the image; wherein, the first center line is the center line in the width direction; and the second center line is the center line in the length direction.

4. The method according to claim 1, characterized in that, The step of simultaneously partitioning the corrected visible light image and the corrected thermal imaging image includes: The corrected visible light image is subjected to foreground segmentation processing to obtain foreground segmentation results, and a first partitioning rule is determined based on the foreground segmentation results; The corrected visible light image and the thermal imaging image are synchronously partitioned according to the first partitioning rule.

5. The method according to claim 1, characterized in that, The step of simultaneously partitioning the corrected visible light image and the corrected thermal imaging image includes: Based on the range of thermal radiation quantization values ​​corresponding to each of the preset multiple temperature levels, the temperature level to which each pixel in the calibrated thermal imaging image belongs is identified. The second partitioning rule is determined based on the temperature level of each pixel in the corrected thermal imaging image. The corrected visible light image and the thermal imaging image are synchronously partitioned according to the second partitioning rule.

6. The method according to claim 1, characterized in that, The step of determining the feature point pair corresponding to the partition based on the sub-region of the corrected visible light image and the sub-region of the corrected thermal imaging image corresponding to the partition includes: Feature point identification is performed on the sub-regions of the corrected visible light image corresponding to the partition to obtain a first set of feature points; Feature point identification is performed on the sub-regions corresponding to the partition in the corrected thermal imaging image to obtain a second set of feature points; Feature point matching is performed on the first set of feature points and the second set of feature points to obtain matching feature point pairs; When the number of matching feature point pairs is greater than or equal to a preset threshold, the matching feature point pair is determined as the feature point pair corresponding to the partition; When the number of matched feature point pairs is less than a preset threshold, a reminder message is output to the user to remind the user to mark the feature point pairs. The matching feature point pair and the user-defined feature point pair are determined as the feature point pair corresponding to the partition.

7. The method according to claim 1, characterized in that, The step of determining the feature point pair corresponding to the partition based on the sub-region of the corrected visible light image and the sub-region of the corrected thermal imaging image corresponding to the partition includes: The corrected visible light image and the corrected thermal imaging image are displayed to the user; In response to the user's annotation operation, the feature point pair consisting of the visible light feature points marked by the user on the sub-region of the corrected visible light image corresponding to the partition, and the thermal infrared feature points corresponding to the visible light feature points marked by the user on the sub-region of the corrected thermal imaging image corresponding to the partition, is determined as the feature point pair corresponding to the partition.

8. The method according to claim 1, characterized in that, The step of performing distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image includes: The initial visible light image is distorted using a pre-determined set of first distortion coefficients corresponding to the visible light image to obtain a distorted visible light image; The initial thermal imaging image is distorted using a pre-determined set of second distortion coefficients corresponding to the thermal infrared image to obtain a rectified thermal infrared image; at least one distortion coefficient in the second set of distortion coefficients is different from that in the first set of distortion coefficients.

9. An image fusion apparatus, characterized in that, The image fusion device includes an acquisition module, a partitioning module, a processing module, and a fusion module; wherein, The acquisition module is used to acquire an initial visible light image and an initial thermal imaging image in the same scene, and to perform distortion correction on the initial visible light image and the initial thermal imaging image respectively to obtain a corrected visible light image and a corrected thermal imaging image; the corrected visible light image and the corrected thermal imaging image have the same resolution; The partitioning module is used to simultaneously perform partitioning processing on the corrected visible light image and the corrected thermal imaging image; The processing module is used to determine the feature point pair corresponding to each partition based on the sub-region of the corrected visible light image corresponding to the partition and the sub-region of the corrected thermal imaging image corresponding to the partition. The processing module is further configured to determine the transformation matrix corresponding to the partition based on the feature point pairs corresponding to the partition, and to register the sub-regions of the corrected visible light image corresponding to each partition based on the transformation matrix corresponding to each partition, so as to obtain the registered visible light image. The fusion module is used to fuse the registered visible light image and the corrected thermal imaging image to obtain a fused image.

10. An image fusion device, characterized in that, The image fusion device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

Images