Dual-light image registration method, program product, medium and apparatus

Abstract

The invention discloses a dual-light image registration method, a program product, a medium and equipment. The method comprises the following steps: acquiring a current infrared image and a current visible light image; obtaining a stored prior homography matrix; calculating a current homography matrix based on the current infrared image, the current visible light image and the prior homography matrix; determining a target homography matrix based on the prior homography matrix and the current homography matrix; and based on the target homography matrix, performing image registration on the current infrared image and the current visible light image to obtain a registered image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a dual - light image registration method, a computer program product, a non - volatile computer storage medium, and a dual - light image registration device. Background Art

[0002] Traditional non - automated dual - light registration methods usually rely on users to complete registration by manually adjusting preset parameters or external auxiliary devices (such as laser ranging). The operation is complex and vulnerable to environmental factors, making it difficult to adapt to dynamic scenarios and complex lighting conditions. Although some existing automatic registration systems reduce human intervention to a certain extent, there are still deficiencies in accuracy and efficiency. For example, some current registration algorithms are not robust enough in high - dynamic scenarios, or are difficult to meet real - time requirements due to high computational complexity. In addition, although binocular vision - based systems have a certain spatial calibration ability, they rely highly on geometric calibration and their performance is greatly reduced in low - light or complex environments, making it difficult to meet the application requirements in multiple scenarios. Summary of the Invention

[0003] To solve the existing technical problems, the present invention provides a dual - light image registration method, a computer program product, a non - volatile computer storage medium, and a dual - light image registration device, which can meet diverse scenario requirements and improve registration accuracy and robustness.

[0004] In a first aspect, a dual - light image registration method is provided, including:

[0005] In a second aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, it implements the dual - light image registration method as described in any item of the first aspect of this application.

[0006] In a third aspect, a dual - light image registration device is provided, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the dual - light image registration method as described in any item of the first aspect of this application.

[0007] In a fourth aspect, a non - volatile computer storage medium is provided, including a computer program. When the computer program is executed by a processor, the processor executes the dual - light image registration method as described in any item of the first aspect of this application.

[0008] This application obtains the current infrared image, the current visible light image, and the prior homography matrix. Based on the current infrared image, the current visible light image, and the prior homography matrix, it calculates the current homography matrix in real time, compares the prior homography matrix with the current homography matrix, and selects a better homography matrix to be applied to the current scene. Therefore, the selected target homography matrix can better adapt to the scene environment and meet diverse scene requirements, thereby improving the registration accuracy and the robustness of registration. Description of the Drawings

[0009] Figure 1 It is a diagram of the application environment of the dual - light image registration method in an embodiment;

[0010] Figure 2 It is another diagram of the application environment of the dual - light image registration method in an embodiment;

[0011] Figure 3 It is a flowchart of the dual - light image registration method in an embodiment;

[0012] Figure 4 It is a working schematic diagram of image fusion based on the dual - light image registration method in an embodiment;

[0013] Figure 5 It is a schematic diagram of a visible light image, an infrared image, and a fused image in an embodiment;

[0014] Figure 6 It is a schematic diagram of the process of calculating the current homography matrix in the dual - light image registration method in an embodiment;

[0015] Figure 7 It is a schematic diagram of multiple feature maps output by the feature extraction model in an embodiment;

[0016] Figure 8 It is a flowchart of determining the current matching pair data corresponding to the current processing layer in the dual - light image registration method in an embodiment;

[0017] Figure 9 It is a schematic diagram of bidirectional matching in an embodiment;

[0018] Figure 10 It is a schematic diagram of the dual - light image registration device in an embodiment;

[0019] Figure 11 It is a schematic diagram of the dual - light image registration device in an embodiment. Detailed Embodiments

[0020] The following further elaborates on the technical solution of the present invention in conjunction with the accompanying drawings of the specification and specific embodiments.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the description of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the scope of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] In the following description, the expression "some embodiments" describes a subset of all possible embodiments. However, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0023] Refer to Figure 1 , refer to Figure 1 , which is an application environment diagram of the dual - light image registration device control method in an embodiment. The dual - light image registration device control method is applied to the dual - light image registration device 10. The dual - light image registration device 10 includes a visible - light image acquisition device 11, an infrared - image acquisition device 12, a processor 13, and a memory 14. The visible - light image acquisition device 11 is used to acquire the current visible - light image of the current scene, and the infrared - image acquisition device 12 is used to acquire the current infrared image of the current scene. Therefore, real - time images can be acquired through the visible - light image acquisition device 11 and the infrared - image acquisition device 12. The processor 13 calculates the real - time current homography matrix according to the acquired current infrared image and current visible - light image, and compares the current unit matrix with the prior homography matrix stored in the memory 14 to determine the target homography matrix, so as to use the target homography matrix to register the current infrared image and the current visible - light image, thereby achieving accurate registration and facilitating the subsequent obtaining of a better dual - light fusion image.

[0024] As Figure 2 shown, Figure 2 , which is another application environment diagram of the dual - light image registration method in an embodiment. The dual - light image registration system in this application environment diagram includes a visible - light image acquisition device 11, an infrared - image acquisition device 12, a processor 13, and a display device 15. The display device 15 can be located in the dual - light image registration device 10 or can be an independent part, a device communicatively connected to the dual - light image registration device 10. The display device 15 is used to display images, for example, to display one or more of the current infrared image, the current visible - light image, and the fusion image.

[0025] The dual - light image registration device 10 includes, but is not limited to, handheld devices, non - handheld autonomous mobile devices, and non - handheld non - autonomous mobile devices. Handheld devices include, but are not limited to, handheld imagers, telescopes, aiming devices, shotguns, etc. Autonomous mobile devices include vehicle devices, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, or robots, etc. Non - handheld non - autonomous mobile devices include, but are not limited to, various types of fixed - mounted electronic devices, etc.

[0026] The visible - light image acquisition device 11 includes a visible - light image sensor and may also be combined with one or more other sensors. The visible - light image acquisition device 11 can be a monocular vision sensor or a multi - ocular vision sensor. For example, it can be a combination of one or more sensors among a visible - light image sensor, a millimeter - wave sensor, a lidar sensor, and a depth sensor. The infrared - light image acquisition device 12 includes an infrared - light image sensor and may also be combined with one or more other sensors. The infrared - light image acquisition device 12 can be a monocular vision sensor or a multi - ocular vision sensor. For example, it can be a combination of one or more sensors among an infrared - light sensor, a millimeter - wave sensor, a lidar sensor, and a depth sensor.

[0027] Among them, the processor 13 can be one or more. When there are multiple processors 13, the multiple processors can be integrated on one chip or independently set on each chip. The dual - light image registration device 10 is a device installed on any type of moving body, such as a vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot, etc. The dual - light image registration device 10 can also be fixedly installed on a certain fixed device or at a fixed position in a fixed scenario.

[0028] The dual - light image registration device 10 may also include other sensor modules, including but not limited to environmental perception sensors and motion attitude sensors. Environmental perception sensors include, but are not limited to, one or more combinations of the following sensors: brightness sensors, temperature sensors, haze sensors, and other environmental sensors. Motion attitude sensors include, but are not limited to, one or more combinations of the following: inertial sensors (Inertial Measurement Unit, IMU), speed sensors, acceleration sensors, gyroscope sensors, geomagnetic sensors, rotation vector sensors, steering wheel angle sensors, level sensors, tilt sensors, vibration sensors, displacement sensors, and gravity sensors, etc.

[0029] With the rapid progress of visual detection technology and the increasing demand for multi-modal perception, the automated registration system for visible and infrared images has become one of the core technologies attracting extensive attention. Compared with the traditional dual-optical image registration method that relies on manual calibration or laser ranging assistance, the automated registration system can achieve a fully automatic, high-precision, and real-time registration and fusion system, significantly solving many defects existing in the current dual-optical registration method.

[0030] The significant differences in characteristics between infrared images and visible light images further increase the technical difficulty of cross-modal registration. Infrared images mainly capture thermal radiation information, lacking the color and detail features of visible light images, and at the same time, their gray value distributions are concentrated and the features are sparse. This modal difference makes it a technical challenge to effectively extract the features of infrared images and accurately match them with visible light images. In addition, the movement of dynamic targets, the changing environmental lighting conditions, and complex backgrounds pose higher challenges to the real-time performance and accuracy of registration algorithms. The above problems are mainly manifested as follows:

[0031] 1. First of all, although the dual-optical registration algorithms based on deep learning have made significant progress in accuracy, especially gradually approaching the human eye level in terms of registration accuracy, these technologies usually require powerful computing resources, which pose a huge challenge to most mobile devices. High-precision algorithms often ignore the real-time requirements, resulting in the inability to meet the needs of real-time performance and low power consumption. In contrast, traditional registration algorithms based on feature point matching and geometric transformation have relatively low computational complexity, but when faced with infrared images and visible light images with large modal differences, their feature extraction capabilities are limited, and it is often difficult to find accurate matching points, resulting in low registration accuracy. In addition, traditional algorithms are usually more sensitive to changes in environmental lighting and target poses, lacking sufficient robustness and prone to failure in dynamic scenes or complex backgrounds. Therefore, how to optimize the computational efficiency and environmental adaptability of the algorithm while ensuring high precision has become the key to solving the real-time performance problem of dual-optical automatic registration.

[0032] 2. The differences between infrared images and visible light images make image registration more complex. Infrared images usually lack the color and detail features in visible light images and mainly rely on contour and gray features, with a large modal difference between the two. And the gray values of infrared images are usually low and concentrated, resulting in relatively weak contrast, which requires the algorithm to have strong feature description and discrimination capabilities during the image registration process. Compared with visible light images, the feature information of infrared images is sparser. How to effectively extract and match these limited features and accurately align them with the rich information in visible light images has become a difficult problem in dual-optical fusion.

[0033] 3. How to improve the adaptability and robustness of the device is another important challenge faced by the dual - light fusion and registration technology. Although advanced sensors can provide clearer image information, how to effectively register and fuse visible - light and infrared images to improve the overall performance of the system remains a key problem in the current technological development.

[0034] In most of the current existing technologies, most dual - light image registration algorithms rely on traditional feature - extraction methods or techniques based on calculating image offsets, and often have defects such as poor registration effect, sensitivity to environmental conditions, and the need for active triggering, making it difficult to achieve fully automatic and real - time dual - light image registration.

[0035] Please refer to Figure 3 , which is a flowchart of the dual - light image registration method provided by an embodiment of this application. The dual - light image registration method is applied to a dual - light image registration device, and the dual - light image registration method includes the following steps:

[0036] S11. Obtain the current infrared image and the current visible - light image.

[0037] In this embodiment, the current infrared image is a real - time infrared image collected by an infrared image acquisition device for the current scene, that is, the current - frame infrared image, and the current visible - light image is a real - time visible - light image collected by a visible - light image acquisition device for the current scene, that is, the current - frame visible - light image. For example, a current infrared image with a resolution of 640x512 and a current visible - light image with a resolution of 1080x1920 are provided to ensure high - quality data input. The infrared image acquisition device and the visible - light image acquisition device are synchronously controlled in time to collect images of the current scene simultaneously, so that it is convenient for the subsequent two types of images to be spatially synchronized, and image registration can be better achieved.

[0038] S12. Obtain the stored prior homography matrix.

[0039] In this embodiment, the homography matrix is used to describe the transformation relationship between the infrared image and the visible - light image. Through the homography matrix, the infrared image can be transformed into the coordinate system of the visible - light image, or the visible - light image can be transformed into the coordinate system of the infrared image. For a dual - light image registration device when it is just out of the factory, the prior homography matrix is the default homography matrix at the factory. During use, or according to the currently collected current infrared image and current visible - light image, the prior homography matrix is updated, so the prior homography matrix is not fixed.

[0040] S13. Calculate the current homography matrix based on the current infrared image, the current visible - light image, and the prior homography matrix.

[0041] In this embodiment, the current homography matrix is a homography matrix calculated in real time, and the current homography matrix changes with the currently captured infrared image and visible light image. In this way, when the scene environment changes, when the currently calculated current homography matrix is better than the prior homography matrix, using the latest calculated current homography matrix can better adapt to the changing scene environment, thereby improving the registration accuracy.

[0042] S14. Determine a target homography matrix based on the prior homography matrix and the current homography matrix.

[0043] In this embodiment, an evaluation index is used to evaluate the prior homography matrix and the current homography matrix. According to the evaluation index value, the target homography matrix to be used in the current frame is determined, that is, the target homography matrix is the homography matrix used in the current frame. In this way, the prior homography matrix and the current homography matrix are compared, and the homography matrix with the optimal evaluation index value among the two is determined as the target homography matrix. This can ensure that a better homography matrix is always used in the current frame, and the prior homography matrix can be continuously updated during the use of the device, thereby ensuring the stability and robustness of the entire device under different environments and conditions.

[0044] S15. Based on the target homography matrix, perform image registration on the current infrared image and the current visible light image to obtain a registered image.

[0045] In this embodiment, using the target homography matrix, the previous infrared image and the current visible light image are transformed to the same coordinate system to achieve image alignment, thereby achieving image registration and obtaining the aligned image, which is the registered image. The dual-light image registration method provided by the embodiments of the present application can automatically perform online dual-light image registration without user intervention, provide accurate registration relationships and meet real-time requirements. For example, in the device, each registration cycle can be controlled within 350 ms, and the fusion frame rate can reach more than 30 frames, realizing real-time dual-light image registration and fusion without user perception, thereby meeting real-time requirements.

[0046] In the above embodiment, the current infrared image, the current visible light image, and the prior homography matrix are obtained. Based on the current infrared image, the current visible light image, and the prior homography matrix, the current homography matrix is calculated in real time, and the prior homography matrix and the current homography matrix are compared, and a better homography matrix is selected and applied to the current scene. Therefore, the selected target homography matrix can better adapt to the scene environment, meet diverse scene requirements, and thereby improve the registration accuracy and the robustness of registration.

[0047] In some embodiments, after obtaining the registered image, the method further includes:

[0048] Obtain a fused image based on the registered image.

[0049] In this embodiment, the registered image is weighted to obtain a fused image. Figure 4 FIG. Figure 4 is a schematic diagram of the operation of image fusion based on a dual - light image registration method in an embodiment. For the specific call logic of the dual - light image registration method and the image fusion algorithm provided in the embodiments of the present application, a two - line concurrent design is adopted. After the current visible - light image and the current infrared image are acquired, they are sent into the program corresponding to the dual - light image registration method. The dual - light image registration method provided in the embodiments of the present application calculates the registration relationship between the current visible - light image and the current infrared image, and continuously updates the registered target homography matrix in the fusion algorithm. In another line, the current visible - light image and the current infrared image are sent into the image fusion algorithm, and the calculated target homography matrix is read to complete the fusion. As Figure 5 shown, Figure 5 FIG. Figure 5 is a schematic diagram of a visible - light image, an infrared image, and a fused image in an embodiment. It can be seen from the example that the fused image is clearer and provides more image details. The dual - light image registration method provided in the embodiments of the present application can provide efficient image registration and fusion capabilities under various complex external conditions, ensuring that the dual - light images can meet the requirements of practical applications in terms of real - time performance, accuracy, and robustness, and greatly improving the reliability and accuracy of environmental perception.

[0050] The dual - light image registration method provided in the embodiments of the present application fully considers the characteristics of infrared images and the requirements of industrial implementation, and can realize the automatic online registration of cross - modal visible light and infrared light on an embedded platform with general computing power. The dual - light fusion network can be well adapted to different embedded platforms and can run in real - time and efficiently on an embedded platform with general computing power.

[0051] In the above - mentioned embodiment, after the dual - light image registration method provided in the embodiments of the present application obtains the aligned registered image, it performs fusion based on the registered image to obtain a fused image, which can provide efficient image registration and fusion capabilities under various complex external conditions, ensuring that the dual - light images can meet the requirements of practical applications in terms of real - time performance, accuracy, and robustness, and greatly improving the reliability and accuracy of environmental perception.

[0052] In some embodiments, Figure 6 FIG. Figure 6 is a schematic diagram of the process of calculating the current homography matrix in the dual - light image registration method in an embodiment;

[0053] S61. Obtain one image from the current infrared image and the current visible - light image as the first single - light image and the other image as the second single - light image, and based on the first single - light image and the prior homography matrix, obtain the transformed image corresponding to the first single - light image.

[0054] In this embodiment, when the first single-light image is the current infrared image, the second single-light image is the current visible light image. When the first single-light image is the current visible light image, the second single-light image is the current infrared image.

[0055] S62. Form input data of the trained feature extraction model based on the second single-light image, and output multiple first feature maps through the feature extraction model. Form input data of the trained feature extraction model based on the transformed image, and output multiple second feature maps through the feature extraction model.

[0056] In this embodiment, the feature extraction model is a pre-trained neural network model. For example, it can be network architectures of different types of backbones. The backbones are very diverse, covering different types of network structures such as Vgg19, ResNet18, ResNet50, R2D2, MOBI LENETV1-V4, DEEPLABV3_MOBI LE, EFFI CI ENTNETV2, etc. For an input image, the feature extraction model outputs feature maps of different sizes at different layers. For the inputs respectively formed by the second single-light image and the transformed image, for the feature maps of the same size, they are output at the same layer. As Figure 7 shown, Figure 7 is a schematic diagram of multiple feature maps output by the feature extraction model in an embodiment; for the same input image, for example, the input image size is 1×3×432×648, where 1 represents the batch, 3 represents the number of channels, and 432×648 represents the image size. Feature map A1 is output at the first layer, with an image size of 1×64×216×324, feature map A2 is output at the second layer, with an image size of 1×64×108×162, feature map A3 is output at the third layer, with an image size of 1×128×54×81, and feature map A4 is output at the fourth layer, with an image size of 1×256×27×41. The feature maps from A1 to A4 are output at different layers. Among them, at the shallow layer, that is, at the layer with a smaller number of layers, features are output, and the output image size is larger and the number of channels is smaller, that is, shallow-layer features are obtained; at the deep layer, that is, at the layer with a larger number of layers, features are output, and the output image size is smaller and the number of channels is larger, that is, deep-layer features are obtained. For example, for a human face, deep-layer features mainly focus on overall features on the face, such as missing an eye and other larger features, while shallow-layer features focus on more delicate features such as the internal texture of the eye or the distribution of eyelashes. The features output at larger layers are more delicate.

[0057] For example, for Figure 7For the input data of the trained feature extraction model formed based on the second single-light image, the feature map A11 corresponding to the second single-light image is output at the first layer number, the feature map A12 corresponding to the second single-light image is output at the second layer number, the feature map A13 corresponding to the second single-light image is output at the third layer number, and the feature map A14 corresponding to the second single-light image is output at the fourth layer number, thereby obtaining four first feature maps. Similarly, for the input data of the trained feature extraction model formed based on the transformed image, the feature map A21 corresponding to the transformed image is output at the first layer number, the feature map A22 corresponding to the transformed image is output at the second layer number, the feature map A23 corresponding to the transformed image is output at the third layer number, and the feature map A24 corresponding to the transformed image is output at the fourth layer number, thereby obtaining four second feature maps.

[0058] The feature extraction model obtains feature maps at different levels. Different-scale bimodal features are extracted layer by layer from the multi-layer feature maps, and different features of the infrared image and the visible light image from coarse to fine are extracted for subsequent hierarchical refinement to complete the matching. The feature extraction model can perform feature matching by extracting multi-layer features from the shallow layer to the deep layer layer by layer, and finally obtain more accurate and finer dual-light image matching point pairs. This can fully exploit the deep regional feature information and shallow detail features in the image, thereby improving the registration accuracy.

[0059] In an optional implementation manner, it is possible to support the feature extraction model to modify different teacher models according to the requirements of the task, that is, the feature extraction model can be a student model. Taking a certain teacher model as an example, the number of parameters of the teacher model is relatively large, reaching more than 100M, while the student model is only 15M, but it can greatly extract the number of correct matching point pairs in the subsequent hierarchical refinement and improve the dual-light registration accuracy. Whether to use different modified teacher models can be determined according to the available computing resources of the current device. In the case of sufficient computing resources, using a refined student model can significantly improve the registration effect of the algorithm. The teacher model is usually a large and complex neural network model that has been fully trained. It performs very well on data, but due to its large size and computational overhead, it may not be suitable for use in resource-constrained environments (such as mobile devices, embedded systems, etc.). The main role of the teacher model is to provide knowledge guidance for the student model. Student model: The student model is a smaller and lighter neural network that improves its performance on data by mimicking the teacher model. The goal of the student model is to "learn" as much as possible from the teacher model about its performance and knowledge, but while maintaining a low computational complexity, it can achieve results as close as possible to the teacher model.

[0060] S63. Based on the multiple first feature maps and the multiple second feature maps, perform multiple matching operations to obtain target matching pair data indicating an accurate match.

[0061] In this embodiment, among multiple first feature maps and multiple second feature maps, feature maps of the same size are output at the same layer number. The first feature maps and the second feature maps at the same layer number are matched to obtain the matching point pair data at this layer number. Then, the positions of the matching point pairs in the matching point pair data at this layer number are mapped to the feature map of the next layer as this layer number to obtain the mapped positions corresponding to the matching point pairs at this layer number. The matching operation of the next layer will refer to the mapped positions corresponding to the matching point pairs at this layer number to perform deeper feature point pair matching.

[0062] Optionally, a current first feature map and a current second feature map at the current processing layer number are obtained from the multiple first feature maps and the multiple second feature maps, and the matching pair data of the previous layer of the current processing layer number is obtained. Based on the current first feature map, the current second feature map, and the matching pair data of the previous layer, a matching operation is performed to determine the current matching pair data corresponding to the current processing layer number. The next layer of the current processing layer number is sequentially updated to the current processing layer number until the current processing layer number is the last layer number.

[0063] Wherein the current processing layer is the layer number corresponding to the feature map where the matching is being performed. The current first feature map and the current second feature map are the feature maps where the matching is being performed.

[0064] For example, taking the above example, the four first feature maps are A11, A12, A13, and A14 respectively, and the four second feature maps are A21, A22, A23, and A24 respectively. Among them, A11 and A21 are output at the same layer number, A12 and A22 are output at the same layer number, A13 and A23 are output at the same layer number, and A14 and A24 are output at the same layer number. First, A11 and A21 are used for feature matching to obtain the matching pair data of the first layer. Then, the matching pair data of the first layer is applied to the second layer. That is, when A12 and S22 are used for feature matching, the matching pair data in the first layer needs to be referred to, and so on iteratively until the feature matching between A14 and A24 is completed.

[0065] In this embodiment, each iteration step starts from the matching features of the current layer. By gradually refining the feature registration layer by layer, a more accurate matching result is finally achieved. In each layer iteration, except for the matching operation of the first layer, the matching operations of other layers will refer to the matching point pair data of the previous layer of the current layer, and each matching point pair is associated with the image features. Then, by mapping the features of the current layer to the previous layer and obtaining the sub-regions corresponding to the matching point pairs of the previous layer in the current layer, new matching relationships are established between the corresponding sub-regions to gradually optimize the registration accuracy.

[0066] In this embodiment, during this multiple matching operation, the feature extraction model also introduces methods such as ratio testing to filter out unreliable matching points and reduce the errors that may accumulate in the previous layers. This hierarchical optimization method has strong generality and is applicable to a variety of different backbone network architectures. Whether it is a traditional convolutional neural network (CNN) or other types of deep learning models, the hierarchical refinement module can effectively improve the accuracy of image registration and provide stable performance in different application scenarios.

[0067] S64. Calculate the current homography matrix based on the target matching pair data.

[0068] In this embodiment, after obtaining the target matching pairs between multiple first feature images and multiple second feature maps, the random sample consensus algorithm (RANSAC) is applied to the target matching pairs to calculate the homography matrix between the visible light and infrared images. RANSAC is an iterative method for estimating the parameters of a mathematical model from data containing a large number of outliers. The core idea of RANSAC is to estimate the model by randomly selecting a data subset, and to screen the matching point pairs through the rationality of the model, and finally find the correct matching points that best conform to the model and calculate the homography matrix between the images.

[0069] In this embodiment, after obtaining the matching point pairs between the transformed image and the second single-light image, the matching point pairs of the transformed image are transformed back to the initial first single-light image through the prior homography matrix to obtain the matching point pairs at the true positions in the single-light image. By applying the hierarchical refinement algorithm from shallow to deep to the feature maps at different levels extracted by the feature extraction model, not only can potential matching points be found within the range from local features to global features, but also a more stable and accurate correspondence can be established between different modality images, laying a solid foundation for subsequent image fusion and analysis.

[0070] In the above embodiment, multiple first feature maps and multiple second feature maps can respectively reflect the visible light image features at different levels and the infrared image features at different levels. Based on the multiple first feature maps and multiple second feature maps, multiple matching operations are performed to obtain the target matching pair data indicating accurate matching. Through features at different levels, in the process of iterative matching layer by layer, from shallow to deep matching, by mapping the features of the current layer to the previous layer and obtaining the sub-region corresponding to the matching point pairs of the previous layer in the current layer, the matching can be further refined in a smaller region. Through this way of layer-by-layer refinement, finer-grained features can be extracted at each layer, and the corresponding relationship between the images can be gradually and accurately aligned, thereby improving the accuracy of image registration.

[0071] In some embodiments, as Figure 8 shown Figure 8It is a flowchart for determining the current matching pair data corresponding to the current processing layer in the dual - light image registration method in an embodiment; performing a matching operation based on the current first feature map, the current second feature map, and the matching pair data of the previous layer, and determining the current matching pair data corresponding to the current processing layer includes:

[0072] S81. For each matching pair in the matching pair data of the previous layer, based on the first point in the matching pair, determine a first search region and the first region feature map corresponding to the first search region in the current first feature map, and based on the second point in the matching pair, determine a second search region and the second region feature map corresponding to the second search region in the current second feature map.

[0073] In this embodiment, the features of each channel in the current first feature map are normalized, that is, the feature values of each channel are divided by the L2 norm of the channel to ensure that the feature vectors of each channel have unit length. Subsequently, the coordinates of the first point in the matching point pair in the current first feature map are enlarged: the coordinates of the first point are doubled, aiming to search for a finer - grained neighborhood (i.e., mapped on the feature map larger than the smallest - sized feature map). Subsequently, a neighborhood (neighbors) is defined, that is, the first search region, for searching for its matching point around each first point. For each point, the function will perform a matching search among multiple neighboring points around it. Similar operations are performed on the current second feature map to obtain the neighborhood corresponding to the second point, that is, the second search region. The first region feature map is the image region corresponding to the first search region in the current first feature map. The second region feature map is the image region corresponding to the second search region in the current second feature map. For example, according to the above example, A11 and A21 are matched to obtain the matching point pair C1 and C2, C1 is located in A11, C2 is located in A21, then the position of C1 is mapped to A12, and the neighborhood corresponding to C1 in A12 is obtained as the first search region, so as to obtain the first region feature map in A12; the position of C2 is mapped to A22, and the neighborhood corresponding to C2 in A22 is obtained as the second search region, so as to obtain the second region feature map in A22. Then, deeper - level matching point pairs are determined in the deeper - level first region feature map and second region feature map, so as to obtain more accurate matching point pairs, and the detailed features corresponding to these more accurate matching point pairs are richer.

[0074] S82. Calculate the region similarity feature data based on the first region feature map and the second region feature map.

[0075] In this embodiment, the first region feature map and the second region feature map are multiplied to obtain a region similarity matrix, that is, the similarity feature data is obtained.

[0076] In this embodiment, a similarity matrix can be obtained by calculating the cosine similarity between the points in the first region feature map and the points in the second region feature map. By measuring the distance between the feature points, it is determined whether they belong to the same physical position. The most commonly used similarity metric is the cosine similarity, which can measure the direction consistency of two feature vectors, rather than just their magnitudes. The higher the value of the cosine similarity, the stronger the similarity between the two feature points, indicating that they are closer in visual features.

[0077] The calculation formula of the cosine similarity is as follows:

[0078]

[0079] where φ represents the cosine similarity, and f A , f B represent the feature vectors of two points respectively.

[0080] The distance d(p A , p B between the feature points is expressed as follows: A , p B ) is as follows:

[0081] d(p A , p B ) = 1 - φ(f A , f B ).

[0082] S83. Based on the regional similarity feature data, determine the target matching point pairs that are bidirectionally matched in the first search region and the second search region, and process each matching pair in turn until all matching pairs are processed, so as to obtain the current matching pair data corresponding to the current processing layer.

[0083] In this embodiment, specifically, in each iteration process, each feature point in the upper layer is mapped to the corresponding area in the lower layer. These areas are usually small blocks in the image (such as 2×2 sub-regions). By processing these small blocks, more accurate matching point pairs can be found. By continuously applying this process, the network adjusts the matching result layer by layer, and finally obtains a more accurate registration relationship. Through multiple iterations, the number of iterations is consistent with the number of the first feature map or the second feature map used in the image. The initial input features come from the shallower layers of the image. The neural network first processes these shallow features to obtain preliminary matching point pairs. Then, these matching points are mapped to the deeper feature maps and further refined in smaller areas. In this way of layer-by-layer refinement, the network can extract finer-grained features at each layer and gradually and accurately align the corresponding relationships between the images. In the final iteration, all matching points are projected onto the feature map of the last layer and further accurate matching is performed at this layer.

[0084] Optionally, determining the target matching point pairs with bidirectional matching in the first search area and the second search area based on the regional similarity feature data includes:

[0085] In each row and each column of the regional similarity feature data, respectively screen the highest similarity score and the second highest similarity score;

[0086] Obtain the target highest similarity score that meets the preset screening conditions and the target position of the target highest similarity score in the regional similarity feature data. The preset screening conditions include: the position where the maximum similarity score in the row indicated by the target position is the target position, the position where the maximum similarity score in the column indicated by the target position is the target position, the ratio of the second highest similarity score to the highest similarity score in the row corresponding to the target position meets the preset ratio range, and the ratio of the second highest similarity score to the highest similarity score in the column corresponding to the target position meets the preset ratio range;

[0087] Generate a target mask corresponding to the target highest similarity score according to the target position;

[0088] Determine the first target matching point in the first regional feature map and the second target matching point in the second regional feature map according to the index corresponding to the target mask, and determine the first target matching point and the second target matching point as the target matching point pairs in the target matching pair data.

[0089] In this embodiment, for a certain point in the first regional feature map, the most similar point in the second regional feature map is searched for, usually the point with the shortest distance. By this method, a potential matching point can be found for each feature point in the first regional feature map. However, one-way matching cannot guarantee the accuracy of the result. To ensure the reliability of the matching, a two-way verification strategy is adopted. This means that for each pair of matching points, the condition of "mutual matching" must be satisfied. For example, given the feature vectors FA and FB extracted from the first regional feature map IA and the second regional feature map IB, where FA corresponds to the feature point pA and FB corresponds to the feature point pB, the nearest neighbor is determined according to the feature distance between pA and pB to identify potential matches. For the feature point pA in the feature map FA, if the distance to pB is the smallest, it is matched with pB. The matching point pair (pA, pB) is only confirmed when they match each other, which means that pA and pB are considered a matching pair only when pB also matches pA.

[0090] In this embodiment, for a position with the highest similarity score of the target that meets the preset screening conditions in the regional similarity feature data, this position is the position with the highest similarity score in its corresponding row and the position with the highest similarity score in its corresponding column, then it indicates that the first feature point in the first regional feature map corresponding to this position and the second feature point in the second regional feature map corresponding to this position are feature points with two-way matching. For example Figure 9 as shown Figure 9 is a schematic diagram of two-way matching in an embodiment. The feature blocks F_A and F_B extracted from A and B will search for the best match of each element in F_A in F_B. The potential match is defined as the nearest neighbor. For a point p_A in F_A, if the ratio of the corresponding scores to the best match p_B and the second-best match p_C is less than a preset ratio, then p_A is matched to p_B. However, this pair of matches will only be accepted when p_B is also matched to p_A. Among them, the best match p_B is the point corresponding to the highest similarity score of the point p_A, and the p_C is the point corresponding to the second-highest similarity score of the point p_A. After finding the corresponding matching points, a target mask mask is saved. The target mask represents the index of the selected matching points in the similarity matrix. For example, in a matrix with a size of 500*300, the point at 200*100 is selected as the matching point, so the value at this point in the target mask is set to 1, indicating that this point is selected as the matching point. After calculating the matching points in this pair of feature maps through the target mask, the position of the first target matching point corresponding to the target mask in the first regional feature map is recorded and the position of the second target matching point corresponding to the target mask in the second regional feature map is recorded.

[0091] In the above embodiments, this two-way matching mechanism effectively avoids incorrect matching and improves the accuracy of matching. In cross-modal image registration, especially in the registration of visible light and infrared light images, the matching quality of feature points directly affects the final registration accuracy. Therefore, this strategy of confirming the match through similarity measurement and two-way verification can significantly improve the accuracy and robustness of the registration result.

[0092] In some embodiments, determining the target homography matrix based on the prior homography matrix and the current homography matrix includes:

[0093] Calculating an evaluation index value based on the current homography matrix or based on the prior homography matrix and the current homography matrix;

[0094] When the evaluation index value indicates that the current homography matrix is better than the prior homography matrix, determining the current homography matrix as the target homography matrix and updating the prior homography matrix to the current homography matrix;

[0095] When the evaluation index value indicates that the current homography matrix is not better than the prior homography matrix, determining the prior homography matrix as the target homography matrix.

[0096] In this embodiment, evaluating the estimated current homography matrix and the prior homography matrix based on the evaluation index value can ensure that the effect of dual-light registration fusion is more robust. When the registration effect of the current homography matrix is worse than that of the prior homography matrix, the prior homography matrix will be used to ensure the fusion effect.

[0097] Optionally, the evaluation index value includes a first mutual information score and a second mutual information score, and / or the number of matching point pairs, where the first mutual information score is calculated based on the prior homography matrix, the second mutual information score is calculated based on the current homography matrix, and the method further includes:

[0098] When the second mutual information score is greater than the first mutual information score, determining that the current homography matrix is better than the prior homography matrix;

[0099] When the second mutual information score is less than or equal to the first mutual information score, determining that the current homography matrix is not better than the prior homography matrix;

[0100] When the number of matching point pairs is less than or equal to a preset number, determining that the current homography matrix is not better than the prior homography matrix;

[0101] When the number of matching point pairs is greater than the preset number, determining that the current homography matrix is better than the prior homography matrix.

[0102] Optionally, calculating the evaluation index value based on the current homography matrix or based on the prior homography matrix and the current homography matrix includes:

[0103] Based on the current infrared image, the current visible light image, and the prior homography matrix, determine a first fused image, select one of the current infrared image and the current visible light image as the evaluation image, calculate a first mutual information score between the evaluation image and the first fused image, and based on the current infrared image, the current visible light image, and the current homography matrix, determine a second fused image, and calculate a second mutual information score between the evaluation image and the second fused image.

[0104] In this embodiment, by comparing the first mutual information score and the second mutual information score, the estimated current homography matrix and the prior homography matrix can be evaluated, that is, the registration and fusion effect of the two can be evaluated. When the registration effect of the current homography matrix is worse than that of the prior homography matrix, the prior homography matrix will be used to ensure the fusion effect.

[0105] Mutual information is a statistic used to measure the similarity between two images. For example, the two images are the evaluation image and the first fused image, or the evaluation image and the second fused image. It is especially suitable for cross-modal image registration. In the image registration task, it is necessary to align images from different sensors or different spectra, such as comparing a visible light image with an infrared image or a fused image. Mutual information quantifies the similarity between two images by calculating the relationship between the gray level or intensity distributions of the images. It can effectively capture the mutual dependence between images. Specifically, mutual information measures the difference between the joint probability distribution of two images and their respective marginal probability distributions. By maximizing the mutual information value, the image alignment effect can be optimized, thereby achieving more accurate image registration. The core calculation formula of mutual information is shown in Formula 3:

[0106]

[0107] In the formula: A represents the evaluation image, B is the first fused image or the second fused image; a ∈ A, b ∈ B represent the pixel gray level values or intensity values in images A and B; p(a, b) represents the joint probability distribution, indicating the probability that pixels a and b appear simultaneously; p(a), p(b) represent the marginal probability distributions, indicating the probabilities of pixels a and b appearing alone.

[0108] In this embodiment, after obtaining the target matching pair data, the number of matching point pairs in the target matching pair data can be calculated. In the dual-light registration algorithm, the number of matching point pairs is also one of the important indicators for measuring the registration accuracy. Matching point pairs are feature points at corresponding positions in two images, which are usually used to calculate the homography matrix or transformation matrix. The larger the number of matching point pairs, the higher the stability and accuracy of image registration usually are, because more matching points can provide more geometric constraints, making the calculation of the homography matrix more reliable. On the contrary, when the number of matching point pairs is small, the estimation error of the homography matrix may increase, resulting in inaccurate image registration results. Therefore, the number of matching point pairs not only reflects the registration accuracy between images, but also reflects the reliability of the registration process. Multi-point matching can reduce the influence of noise, improve the robustness of calculation, and make the finally obtained transformation matrix more accurate. Especially when facing complex cross-modal registration tasks, a sufficient number of matching point pairs can effectively reduce the influence of false matches, thus ensuring the quality and reliability of image registration. In an alternative manner, a comprehensive score can be obtained based on a preset weight, the first mutual information score, the second mutual information score, and the number of matching point pairs. When the comprehensive score is greater than the preset comprehensive score, it indicates that the current homography matrix is better than the prior matrix. By comprehensively using both to judge the accuracy difference between the homography matrix calculated by the algorithm and the prior homography matrix, the dual-light registration fusion device can obtain a better effect.

[0109] In the above embodiment, during the registration process, evaluation indicators can be used to evaluate the registration result in real time. Only when the registration effect is better than the prior relationship will the result be applied to the subsequent dual-light image fusion unit, thereby ensuring the stability and robustness of the entire system in different environments and conditions, effectively avoiding the situation of registration failure, and ensuring the registration accuracy.

[0110] In some embodiments, most of the post-processing part in the dual-light image registration method provided by at least one embodiment of the present application is completed based on the CPU. However, considering the real-time performance of the algorithm and the resource occupancy problem, the pre-processing and hierarchical refinement part of the dual-light image registration method provided by at least one embodiment of the present application are integrated into a part of the model and GPU operation is adopted, which can greatly improve the calculation speed. And in a chip with relatively sufficient resources, concurrent calculation methods can be adopted, which can greatly reduce the calculation time of the algorithm, complete some steps of hierarchical refinement on the feature map obtained by the feature extraction model, quickly extract the matching point pairs, and complete the registration of the dual-light images.

[0111] Combined with one or more of the above embodiments, the technical solution of the present application has at least the following characteristics: At least one embodiment of the present application is based on the prior registration relationship and the two-stage optimization strategy, and successfully breaks through the bottlenecks of traditional registration technology and high-precision AI registration technology. By extracting complementary features of two modalities (visible light image and infrared image) through a deep learning network and combining a dynamic evaluation mechanism, it is ensured that the registration result has high precision and high robustness. At the same time, the method has been comprehensively optimized and can achieve a registration cycle of less than 350 milliseconds per time and a fusion frame rate of more than 30 frames per second under the condition of limited computing resources, so as to meet the real-time requirements.

[0112] On the other hand, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the dual-light image registration method described in any embodiment of the present application.

[0113] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program for implementing each step of the dual-light image registration method can be a dual-light image registration device.

[0114] Please refer to Figure 10 , an embodiment of the present application provides a dual-light image registration device, including: an acquisition module 101, configured to acquire a current infrared image and a current visible light image; the acquisition module 101 is further configured to acquire a stored prior homography matrix; a calculation module 102, configured to calculate a current homography matrix based on the current infrared image, the current visible light image, and the prior homography matrix; a determination module 103, configured to determine a target homography matrix based on the prior homography matrix and the current homography matrix; a registration fusion module 104, configured to perform image registration on the current infrared image and the current visible light image based on the target homography matrix to obtain a registered image.

[0115] Optionally, the registration fusion module 104 is further configured to:

[0116] Based on the registered image, obtain a fused image.

[0117] Optionally, the calculation module 102 is further configured to:

[0118] Obtain one image from the current infrared image and the current visible light image as a first single-light image and the other image as a second single-light image, and based on the first single-light image and the prior homography matrix, obtain a transformed image corresponding to the first single-light image;

[0119] Based on the second single-light image, form input data of a trained feature extraction model, and output a plurality of first feature maps through the feature extraction model, and based on the transformed image, form input data of the trained feature extraction model, and output a plurality of second feature maps through the feature extraction model;

[0120] Based on the multiple first feature maps and the multiple second feature maps, perform multiple matching operations to obtain target matching pair data indicating exact matching;

[0121] Calculate the current homography matrix based on the target matching pair data.

[0122] Optionally, the determination module 103 is further configured to:

[0123] Calculate an evaluation index value based on the current homography matrix or based on the prior homography matrix and the current homography matrix;

[0124] When the evaluation index value indicates that the current homography matrix is better than the prior homography matrix, determine the current homography matrix as the target homography matrix, and update the prior homography matrix to the current homography matrix;

[0125] When the evaluation index value indicates that the current homography matrix is not better than the prior homography matrix, determine the prior homography matrix as the target homography matrix.

[0126] Optionally, the evaluation index value includes a first mutual information score and a second mutual information score, and / or the number of matching point pairs. The determination module 103 is further configured to:

[0127] When the second mutual information score is greater than the first mutual information score, determine that the current homography matrix is better than the prior homography matrix;

[0128] When the second mutual information score is less than or equal to the first mutual information score, determine that the current homography matrix is not better than the prior homography matrix;

[0129] When the number of matching point pairs is less than or equal to a preset number, determine that the current homography matrix is not better than the prior homography matrix;

[0130] When the number of matching point pairs is greater than the preset number, determine that the current homography matrix is better than the prior homography matrix.

[0131] Optionally, the calculation module 102 is further configured to:

[0132] Based on the current infrared image, the current visible light image, and the prior homography matrix, determine a first fused image, select one of the current infrared image and the current visible light image as an evaluation image, calculate a first mutual information score between the evaluation image and the first fused image, and based on the current infrared image, the current visible light image, and the current homography matrix, determine a second fused image, and calculate a second mutual information score between the evaluation image and the second fused image.

[0133] Optionally, the calculation module 102 is further configured to:

[0134] Obtain the current first feature map and the current second feature map at the current processing layer from the multiple first feature maps and the multiple second feature maps, and obtain the matching pair data of the previous layer of the current processing layer. Based on the current first feature map, the current second feature map, and the matching pair data of the previous layer, perform a matching operation to determine the current matching pair data corresponding to the current processing layer, and sequentially update the next layer of the current processing layer to the current processing layer until the current processing layer is the last layer.

[0135] Optionally, the calculation module 102 is further configured to:

[0136] For each matching pair in the matching pair data of the previous layer, based on the first point in the matching pair, determine a first search region and a first region feature map corresponding to the first search region in the current first feature map, and based on the second point in the matching pair, determine a second search region and a second region feature map corresponding to the second search region in the current second feature map;

[0137] Calculate region similarity feature data based on the first region feature map and the second region feature map;

[0138] Based on the region similarity feature data, determine the target matching point pairs that are bidirectionally matched in the first search region and the second search region, and process each matching pair in sequence until all matching pairs are processed to obtain the current matching pair data corresponding to the current processing layer.

[0139] Optionally, the calculation module 102 is further configured to:

[0140] In each row and each column of the region similarity feature data, respectively screen the highest similarity score and the second highest similarity score;

[0141] Obtain the target highest similarity score that meets the preset screening conditions and the target position of the target highest similarity score in the region similarity feature data. The preset screening conditions include: the position where the maximum similarity score in the row indicated by the target position is the target position, the position where the maximum similarity score in the column indicated by the target position is the target position, the ratio of the second highest similarity score to the highest similarity score in the row corresponding to the target position meets the preset ratio range, and the ratio of the second highest similarity score to the highest similarity score in the column corresponding to the target position meets the preset ratio range;

[0142] Generate a target mask corresponding to the target highest similarity score according to the target position.

[0143] Determine a first target matching point in the first regional feature map and a second target matching point in the second regional feature map according to the index corresponding to the target mask, and determine the first target matching point and the second target matching point as the target matching point pair in the target matching pair data.

[0144] Those skilled in the art can understand that Figure 10 the structure of the dual-light image registration device in [] does not constitute a limitation on the dual-light image registration device, and each of the above modules can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the dual-light image registration device in hardware form or independent of it, or can be stored in the memory of the dual-light image registration device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules. In other embodiments, the dual-light image registration device may include more or fewer modules than shown in the figure.

[0145] Please refer to Figure 11 , on the other hand, an embodiment of the present application further provides a dual-light image registration device 10, including a processor 13 and a memory 14. The memory 14 stores a computer program. When the computer program is executed by the processor, the processor 13 is caused to execute the steps of the dual-light image registration method provided in any one of the above embodiments of the present application.

[0146] Among them, the processor 13 is a control center, which connects various parts of the entire dual-light image registration device through various interfaces and lines, and executes various functions of the dual-light image registration device and processes data by running or executing software programs and / or modules stored in the memory 14, and calling data stored in the memory 14. Optionally, the processor 13 may include one or more processing cores; preferably, the processor 13 may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, user pages, and application programs, etc., and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above modulation and demodulation processor may not be integrated into the processor 13.

[0147] The memory 14 can be used to store software programs and modules. The processor 13 executes various functional applications and data processing by running the software programs and modules stored in the memory 14. The memory 14 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the dual-light image registration device, etc. In addition, the memory 14 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 14 can also include a memory processor to provide the processor 13 with access to the memory 14.

[0148] In another aspect of the embodiments of the present application, a non-volatile computer storage medium is further provided, storing a computer program, which when executed by a processor, causes the processor to execute the steps of the dual-light image registration method provided in any of the above embodiments of the present application.

[0149] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods provided in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims described above.

Claims

1. A bi-optical image registration method, characterized in that: include: Obtain the current infrared image and the current visible light image; Get the stored prior homography matrix; Calculating a current homography matrix based on the current infrared image, the current visible light image and the prior homography matrix; Determine a target homography matrix based on the prior homography matrix and the current homography matrix; Based on the target homography matrix, image registration is performed on the current infrared image and the current visible light image to obtain a registered image.

2. The bi-optical image registration method according to claim 1, characterized in that: After obtaining the registration image, the method further includes: Based on the registered images, a fused image is obtained.

3. The bi-optical image registration method according to claim 1, wherein: The calculating of the current homography matrix based on the current infrared image, the current visible light image and the prior homography matrix comprises: Acquire one image as a first single light image and another image as a second single light image from the current infrared image and the current visible light image, and obtain a transformed image corresponding to the first single light image based on the first single light image and the prior homography matrix; forming input data of a trained feature extraction model based on the second single light image, and outputting a plurality of first feature maps through the feature extraction model, forming input data of a trained feature extraction model based on the transformed image, and outputting a plurality of second feature maps through the feature extraction model; Based on the plurality of the first feature graphs and the plurality of the second feature graphs, performing a plurality of matching operations to obtain target matching pair data indicating an exact match; The current homography matrix is ​​calculated based on the target matching pair data.

4. The bi-optical image registration method according to claim 1, wherein: The determining of the target homography matrix based on the prior homography matrix and the current homography matrix comprises: Calculating an evaluation index value based on the current homography matrix or based on the prior homography matrix and the current homography matrix; When the evaluation index value indicates that the current homography matrix is ​​better than the priori homography matrix, determining the current homography matrix as the target homography matrix, and updating the priori homography matrix to the current homography matrix; When the evaluation index value indicates that the current homography matrix is ​​not better than the priori homography matrix, the priori homography matrix is ​​determined as the target homography matrix.

5. The bi-optical image registration method according to claim 4, characterized in that: The evaluation index value includes a first mutual information score and a second mutual information score, and / or the number of matching point pairs, wherein the first mutual information score is calculated based on a priori homography matrix, and the second mutual information score is calculated based on a current homography matrix, and the method further includes: When the second mutual information score is greater than the first mutual information score, determining that the current homography matrix is ​​better than the prior homography matrix; When the second mutual information score is less than or equal to the first mutual information score, determining that the current homography matrix is ​​not better than the prior homography matrix; When the number of matching point pairs is less than or equal to a preset number, determining that the current homography matrix is ​​not better than the prior homography matrix; When the number of the matching point pairs is greater than a preset number, it is determined that the current homography matrix is ​​superior to the priori homography matrix.

6. The bi-optical image registration method according to claim 5, characterized in that: The calculating of the evaluation index value based on the current homography matrix or based on the prior homography matrix and the current homography matrix comprises: Based on the current infrared image, the current visible light image and the prior homography matrix, a first fused image is determined, one of the current infrared image and the current visible light image is selected as an evaluation image, and a first mutual information score between the evaluation image and the first fused image is calculated; and based on the current infrared image, the current visible light image and the current homography matrix, a second fused image is determined, and a second mutual information score between the evaluation image and the second fused image is calculated.

7. The bi-optical image registration method according to claim 3, characterized in that: The performing multiple matching operations based on the multiple first feature graphs and the multiple second feature graphs to obtain target matching pair data indicating an exact match includes: Obtain the current first feature map and the current second feature map under the current processing layer number from multiple first feature maps and multiple second feature maps, and obtain the matching pair data of the previous layer of the current processing layer number; based on the current first feature map, the current second feature map and the matching pair data of the previous layer, perform a matching operation to determine the current matching pair data corresponding to the current processing layer number, and update the next layer of the current processing layer number to the current processing layer number in sequence until the current processing layer number is the last layer number.

8. The bi-optical image registration method according to claim 7, characterized in that: The performing a matching operation based on the current first feature map, the current second feature map and the matching pair data of the previous layer to determine the current matching pair data corresponding to the current processing layer number includes: For each matching pair in the matching pair data of the previous layer, based on the first point in the matching pair, determine a first search area and a first area feature map corresponding to the first search area in the current first feature map, and based on the second point in the matching pair, determine a second search area and a second area feature map corresponding to the second search area in the current second feature map; Calculating regional similarity feature data based on the first regional feature map and the second regional feature map; Based on the regional similarity feature data, target matching point pairs for bidirectional matching in the first search area and the second search area are determined, and each matching pair is processed in turn until all matching pairs are processed to obtain current matching pair data corresponding to the current processing layer number.

9. The bi-optical image registration method according to claim 8, characterized in that: The determining, based on the region similarity feature data, target matching point pairs for bidirectional matching in the first search region and the second search region comprises: In each row and each column of the regional similarity feature data, respectively select the highest similarity score and the second highest similarity score; Obtaining a target highest similarity score that meets a preset screening condition and a target position of the target highest similarity score in the regional similarity feature data, wherein the preset screening condition includes: the position of the maximum similarity score in the row indicated by the target position is the target position, the position of the maximum similarity score in the column indicated by the target position is the target position, the ratio of the second highest similarity score to the highest similarity score corresponding to the row indicated by the target position meets a preset ratio range, and the ratio of the second highest similarity score to the highest similarity score corresponding to the column indicated by the target position meets a preset ratio range; According to the target position, generating a target mask corresponding to the highest similarity score of the target; According to the index corresponding to the target mask, a first target matching point is determined in the first region feature map and a second target matching point is determined in the second region feature map, and the first target matching point and the second target matching point are determined as a target matching point pair in the target matching pair data.

10. A bi-optical image registration device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the bi-optical image registration method according to any one of claims 1 to 9.

11. A non-volatile computer storage medium comprising a computer program, characterized in that A computer program is stored, and when the computer program is executed by a processor, the processor executes the bi-optical image registration method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to execute the bi-optical image registration method according to any one of claims 1 to 9.

Images