Cross-modal image registration method based on deep learning, medium, device and program product
Through a cross-modal image registration method based on deep learning, the network model of homography matrix estimation solves the accuracy and real-time problems of traditional methods in non-ideal situations, achieving higher image registration robustness and Timeliness.
Patent Information
- Application Number
- CN202411132822.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
AI Technical Summary
The traditional joint calibration-based image registration method is difficult to achieve better results under non-ideal situations such as repeated textures, weak textures or no textures, large foreground interference, night scenes or noise interference, resulting in the accuracy of cross-modal image registration. and real-time issues.
Using a cross-modal image registration method based on deep learning, a homography matrix is obtained through camera joint calibration, a homography matrix estimation network model is built, and the training data set is made using the registered infrared visible light public data set, trained and deployed. Homography matrix estimation network model to achieve cross-modal image registration.
This method can extract rich image features in an end-to-end manner, improve the robustness and timeliness of image registration, overcome the difficulties of traditional methods in non-ideal situations, and achieve higher accuracy and real-time.
Smart Images

Figure CN119090925A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image registration, and in particular to a cross-modal image registration method, medium, device and program product based on deep learning. Background Art
[0002] Image registration is a typical problem and technical difficulty in the field of image processing research. Its purpose is to compare or fuse images of the same object acquired under different conditions. As an important and basic problem in the field of image processing, it has a wide range of application scenarios. For example, images may come from different acquisition devices, taken at different times, and shot from different angles, etc.
[0003] Traditional registration methods based on joint calibration have difficulty in achieving good results in non-ideal situations such as repeated textures, weak textures or no textures, large foreground interference, night scenes or noise interference. Summary of the invention
[0004] The present invention aims to provide a cross-modal image registration method, medium, device and program product based on deep learning to solve the accuracy and real-time problems of cross-modal image registration and overcome the difficulties faced by traditional joint calibration-based registration methods in non-ideal conditions such as repeated textures, weak textures or no textures, large foreground interference, night scenes or noise interference.
[0005] The present invention provides a cross-modal image registration method based on deep learning, comprising the following steps:
[0006] a. Obtain the homography matrix of the application scenario as a reference through the camera joint calibration method;
[0007] b. Based on the registered infrared and visible light public dataset, create a training dataset in combination with the homography matrix in step a;
[0008] c. Build a homography matrix estimation network model based on deep learning;
[0009] d. Complete the training of the homography matrix estimation network model;
[0010] e. Deploy the trained homography matrix estimation network model to the application scenario to realize cross-modal image registration.
[0011] Furthermore, step a includes the following sub-steps:
[0012] a1. According to the optical-mechanical structure and shooting distance of the infrared and visible light imaging system, the field of view difference between the corresponding infrared image and the visible light image, i.e. the homography matrix, is determined;
[0013] a2. The homography matrix is given by the joint calibration method and converted into the offset of the two-dimensional coordinates of the four corner points. That is, the homography matrix is equivalent to the representation of 8 scalars.
[0014] Furthermore, step b includes the following sub-steps:
[0015] b1. Based on any pair of infrared and visible light images in the registered infrared and visible light public dataset, select the same rectangular area, where the visible light image intercepts the visible light image block within the rectangular area, and the infrared image marks the coordinates of the four fixed vertices of the rectangular area;
[0016] b2. Based on the reference homography matrix, after the infrared image is transformed by homography, the infrared image block is intercepted according to the coordinates of the four fixed vertices in the rectangular area;
[0017] b3. Pairing the visible light image block and the infrared image block, and using the homography matrix as a label to construct a labeled data;
[0018] b4. Introduce weak random noise into the homography matrix while ensuring that its homography properties remain unchanged, repeat steps b1 to b3 to construct labeled data until the sample size of the training data set meets the requirements, and the construction of the training data set is completed.
[0019] Preferably, the length and width of the rectangular area are integer multiples of 32 by default.
[0020] Preferably, the sample size of the training data set is not less than 10,000 examples.
[0021] Furthermore, in step c, the feature extraction structure of the homography matrix estimation network model is designed according to the VGG network structure; the input of the homography matrix estimation network model is the infrared image block and the visible light image block, and the output is an estimate of the representation of the homography matrix equivalent to 8 scalars.
[0022] Furthermore, in step d, the training data set is divided into a training set, a validation set and a test set according to a certain ratio, and the accuracy and timeliness of the network model are supervised according to the performance of the validation set and the test set during training.
[0023] The present invention also provides a computer terminal storage medium storing computer terminal executable instructions, wherein the computer terminal executable instructions are used to execute the described deep learning-based cross-modal image registration method.
[0024] The present invention also provides a computing device, comprising:
[0025] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned deep learning-based cross-modal image registration method.
[0026] The present invention also provides a computer program product, which implements the above-mentioned deep learning-based cross-modal image registration method when executed by a processor.
[0027] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0028] The present invention solves the accuracy and real-time problems of cross-modal image registration through a cross-modal image registration method based on deep learning; overcomes the difficulties faced by traditional registration methods based on joint calibration under non-ideal conditions such as repeated textures, weak textures or no textures, large foreground interference, night scenes or noise interference; can extract sufficiently rich image features to complete image registration in an end-to-end manner; unlike traditional methods that are heavily dependent on the quality of feature points, this method has better robustness in expressing image features; at the same time, the end-to-end image registration method is superior to the traditional multi-step serial method to achieve registration in terms of timeliness, and well solves the challenges of cross-modal image registration tasks in practical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 Flow chart of a cross-modal image registration method based on deep learning in an embodiment of the present invention.
[0030] Figure 2 A schematic diagram of preparing a training data set in an embodiment of the present invention.
[0031] Figure 3 Schematic diagram of a homography matrix estimation network model in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] Example
[0035] like Figure 1 As shown, this embodiment proposes a cross-modal image registration method based on deep learning, comprising the following steps:
[0036] a. Obtain the homography matrix of the application scenario as a reference through the camera joint calibration method:
[0037] a1. For a specific optical-mechanical structure and shooting distance of an infrared-visible light imaging system, the field of view difference between the corresponding infrared image and the visible light image, i.e., the homography matrix, is determined.
[0038] a2. Through the joint calibration method, the homography matrix given can be simply converted into the offset of the two-dimensional coordinates of the four corner points through the formula, that is, the homography matrix can be equivalent to the representation of 8 scalars.
[0039] b. Based on the registered infrared and visible light public dataset, combine the homography matrix in step a to create a training dataset:
[0040] b1. Based on any pair of infrared and visible light images in the registered infrared and visible light public dataset, select the same rectangular area, where the visible light image intercepts the visible light image block within the rectangular area, and the infrared image marks the coordinates of the four fixed vertices of the rectangular area. The length and width of the rectangular area are integer multiples of 32 by default. That is:
[0041] H=32*n,W=32*m
[0042] Wherein, H is the height of the rectangular area, W is the width of the rectangular area, and n and m are positive integers;
[0043] b2. Based on the reference homography matrix, after the infrared image undergoes homography transformation, the infrared image block is intercepted according to the coordinates of the four fixed vertices in the rectangular area. The specific process is as follows: Figure 2 As shown;
[0044] b3. The above visible light image blocks and infrared image blocks form paired data and homography matrices as labels to construct a labeled data set;
[0045] b4. Introduce weak random noise into the homography matrix while ensuring that its homography properties remain unchanged, repeat steps b1 to b3 to construct labeled data until the sample size of the training data set meets the requirements, generally not less than 10,000 cases, and complete the construction of the training data set.
[0046] c. Build a homography matrix estimation network model based on deep learning. The specific structure of the homography matrix estimation network model is as follows: Figure 3 As shown in the figure, its feature extraction structure is designed according to the VGG network structure. The input of the homography matrix estimation network model is the infrared image block and the visible light image block, the size of which is H*W*3 and H*W*1 respectively, and the output is the estimate of the homography matrix equivalent to the representation of 8 scalars, the size of which is a one-dimensional vector containing 8 components.
[0047] d. Complete the training of the homography matrix estimation network model; in this embodiment, the training data set is divided into a training set, a validation set, and a test set according to an 8:1:1 ratio, and the accuracy and timeliness of the network model are supervised according to the performance of the validation set and the test set during training.
[0048] e. Deploy the trained homography matrix estimation network model to the application scenario to realize cross-modal image registration.
[0049] In addition, in some embodiments, a computer terminal storage medium is proposed, which stores computer terminal executable instructions, and the computer terminal executable instructions are used to execute the cross-modal image registration method based on deep learning as described in the previous embodiments. Examples of computer storage media include magnetic storage media (e.g., floppy disks, hard disks, etc.), optical recording media (e.g., CD-ROMs, DVDs, etc.) or memories, such as memory cards, ROMs, or RAMs. Computer storage media can also be distributed on network-connected computer systems, such as application stores.
[0050] In addition, in some embodiments, a computing device is proposed, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the cross-modal image registration method based on deep learning as described in the above embodiments. Examples of computing devices include PCs, tablet computers, smart phones, or PDAs.
[0051] In addition, in some embodiments, a computer program product is proposed, and when the computer program is executed by a processor, the above-mentioned deep learning-based cross-modal image registration method is implemented.
[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A cross-modal image registration method based on deep learning, characterized in that: The steps include: a. Obtain the homography matrix of the application scenario as a reference through the camera joint calibration method; b. Based on the registered infrared and visible light public dataset, create a training dataset in combination with the homography matrix in step a; c. Build a homography matrix estimation network model based on deep learning; d. Complete the training of the homography matrix estimation network model; e. Deploy the trained homography matrix estimation network model to the application scenario to realize cross-modal image registration.
2. The cross-modal image registration method based on deep learning according to claim 1, characterized in that: Step a includes the following sub-steps: a1. According to the optical-mechanical structure and shooting distance of the infrared and visible light imaging system, the field of view difference between the corresponding infrared image and the visible light image, i.e. the homography matrix, is determined; a2. The homography matrix is given by the joint calibration method and converted into the offset of the two-dimensional coordinates of the four corner points. That is, the homography matrix is equivalent to the representation of 8 scalars.
3. The cross-modal image registration method based on deep learning according to claim 2, characterized in that: Step b includes the following sub-steps: b1. Based on any pair of infrared and visible light images in the registered infrared and visible light public dataset, select the same rectangular area, where the visible light image intercepts the visible light image block within the rectangular area, and the infrared image marks the coordinates of the four fixed vertices of the rectangular area; b2. Based on the reference homography matrix, after the infrared image is transformed by homography, the infrared image block is intercepted according to the coordinates of the four fixed vertices in the rectangular area; b3. The visible light image block and the infrared image block form paired data, and the homography matrix is used as a label to construct a labeled data; b4. Introduce weak random noise into the homography matrix while ensuring that its homography properties remain unchanged, repeat steps b1 to b3 to construct labeled data until the sample size of the training data set meets the requirements, and the construction of the training data set is completed.
4. The cross-modal image registration method based on deep learning according to claim 3, characterized in that: The length and width of the rectangular area are integer multiples of 32 by default.
5. The cross-modal image registration method based on deep learning according to claim 3, characterized in that: The sample size of the training data set is not less than 10,000 examples.
6. The cross-modal image registration method based on deep learning according to claim 1, characterized in that: In step c, the feature extraction structure of the homography matrix estimation network model is designed according to the VGG network structure; the input of the homography matrix estimation network model is the infrared image block and the visible light image block, and the output is an estimate of the representation of the homography matrix equivalent to 8 scalars.
7. The cross-modal image registration method based on deep learning according to claim 1, characterized in that: In step d, the training data set is divided into a training set, a validation set and a test set according to a certain ratio, and the accuracy and timeliness of the network model are supervised according to the performance of the validation set and the test set during training.
8. A computer terminal storage medium storing computer terminal executable instructions, characterized in that: The computer terminal executable instructions are used to execute the deep learning-based cross-modal image registration method as described in any one of claims 1-7.
9. A computing device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based cross-modal image registration method as described in any one of claims 1-7.
10. A computer program product, characterized in that When the computer program is executed by a processor, it implements the deep learning-based cross-modal image registration method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Lightweight image registration method, system and device based on global homography estimation
CN114565511A
Thermal infrared and visible light image synchronous registration method based on time-space unification
CN114897947A
Progressive unmanned aerial vehicle visible light thermal infrared image registration method
CN116912298A
