Image fusion method, device and electronic medium for joint image registration

Abstract

The present application relates to the field of autonomous driving technology, specifically, the present invention relates to an image fusion method, device and electronic device for joint image registration, which belongs to the field of autonomous driving technology. Among them, the method uses a deep learning network to construct an image registration and fusion model, and performs registration and fusion on unregistered infrared / visible light images to achieve rapid perception of the vehicle environment. The model first extracts features from the image, then performs feature matching and calibration on the extracted feature information, and finally obtains the fused image through the reconstruction module. In the network training stage, the information loss of the fused image and the infrared / visible light image is calculated, and the model is trained based on the loss to obtain a fully trained network model. The present application uses an image registration fusion end-to-end network model to comprehensively utilize the advantages of visible light and infrared images to quickly obtain environmental information, so that the model can be applied to scenarios with high real-time requirements for vehicle environmental perception.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an image fusion method, device, and electronic medium for joint image registration. Background Art

[0002] In recent years, image fusion technology, as a major branch of multi-sensor information fusion, has garnered widespread attention. Image fusion integrates image information from different sensors through algorithms to generate information-rich images that align with human visual perception. This technology provides more comprehensive image information for subsequent engineering systems, assisting them in making more accurate decisions and becoming an indispensable component of engineering decision-making.

[0003] Existing image fusion methods usually only focus on the image fusion algorithm itself, assuming that the source images have been pixel-level registered. Directly fusing images that are not accurately registered will lead to ghosting, blurring, and other phenomena in the fusion results. Therefore, existing image fusion methods largely ignore the importance of image registration. In actual engineering applications, a semi-automatic operation mode with manual intervention is usually used to achieve image registration, and then the registered images are fused, which greatly reduces the efficiency of the image processing task. In addition, there are differences in imaging mechanisms between cross-modal images, making it difficult to capture feature information for registration. Therefore, general image registration algorithms are difficult to apply to multimodal images with different features. Therefore, how to solve the problem of accurate registration of multimodal image fusion in engineering applications is a major challenge for multimodal image fusion technology.

[0004] With the rapid development of intelligent transportation systems, vehicle environment perception technology has become a core component of autonomous driving and advanced driver assistance systems. This technology integrates multiple sensors (such as radar, lidar, and cameras) to obtain real-time information about the vehicle's surroundings, enabling functions such as path planning, obstacle detection, and collision warning.

[0005] Existing vehicle environmental perception technology primarily relies on visible light vision sensors (Tesla) to acquire real-time information about the vehicle's surroundings. However, single visible light vision sensors are susceptible to interference from external environmental factors, resulting in inaccurate environmental information acquisition and poor recognition accuracy for the vehicle's environmental perception system. In combination with infrared and visible light image fusion technology, it is possible to simultaneously acquire information about the vehicle's surroundings and thermal radiation, enhancing the vehicle's environmental perception capabilities. Summary of the Invention

[0006] In view of this, it is necessary to provide an image fusion method, device, and electronic device for joint image registration. This method addresses the problem that existing technologies are not directly applicable to applications requiring high real-time performance. While ensuring real-time performance, it also addresses the problem in vehicle environmental perception systems where a single visible light vision sensor is susceptible to interference from external environmental factors, resulting in poor recognition accuracy.

[0007] In order to solve the above problems, the present application provides an image fusion method for joint image registration, comprising:

[0008] An initial image registration and fusion model is constructed using a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence;

[0009] Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image;

[0010] Calculating the loss between the fused image and the visible light image and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model;

[0011] Image fusion is performed on the visible light image and the infrared image to be measured based on the fully trained image registration and fusion model.

[0012] In some possible implementations, inputting several preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image includes:

[0013] Inputting the visible light image and the infrared image into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features;

[0014] Performing feature point matching on initial features of the visible light image and the infrared image based on the feature matching module to obtain geometric deformation parameters, and registering the initial features based on the geometric deformation parameters to obtain registration features;

[0015] Based on the image reconstruction module, feature fusion and feature reconstruction are performed on the registration features to obtain fusion features.

[0016] In some possible implementations, the feature extraction module is formed by stacking several convolution units.

[0017] In some possible implementations, the feature extraction module includes ten convolutional units connected in sequence, wherein the first, third and ninth convolutional units each include a convolutional layer, a normalization layer and an excitation layer connected in sequence, the second and fourth convolutional units each include a convolutional layer, a normalization layer, an excitation layer and a maximum pooling layer connected in sequence, the fifth and seventh convolutional units each include an upsampling layer, a convolutional layer and an excitation layer connected in sequence, the sixth and eighth convolutional units include a splicing layer, a convolutional layer, a normalization layer and an excitation layer connected in sequence, the tenth convolutional unit includes a convolutional layer, the output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit.

[0018] In some possible implementations, the normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

[0019] In some possible implementations, the loss function used when training the initial image registration fusion model is:

[0020]

[0021] Where, represents the intensity information loss function of the fused image, the visible light image, and the infrared image, Represents the gradient information loss function of the fused image, visible light image and infrared image, Indicates a trade-off and The weight parameter of .

[0022] In some possible implementations, the intensity information loss function and the gradient information loss function are specifically:

[0023]

[0024]

[0025] Where, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image, Indicates that the similarity between the true matrix and the estimated matrix is measured by calculating the L2 norm. Represents the mean pixel value of the matrix. Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

[0026] In some possible implementations, the preset sets of visible light images and infrared images are obtained by the following method:

[0027] Acquire several sets of visible light images and infrared images;

[0028] Grayscale conversion, denoising and correction are performed on the groups of visible light images and infrared images to obtain the pre-processed groups of visible light images and infrared images.

[0029] The present application also provides an image fusion device for joint image registration, comprising:

[0030] A model construction unit is used to construct an initial image registration and fusion model based on a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence;

[0031] a model training unit, configured to input a plurality of preset sets of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, calculate information loss between the fused image and the visible light image and the infrared image, and train the initial image registration and fusion model based on the information loss to obtain a fully trained image registration and fusion model;

[0032] The image fusion unit is used to fuse the visible light image and the infrared image to be measured based on the trained image registration and fusion model.

[0033] The present application also provides an electronic device, comprising a memory and a processor;

[0034] The memory is used to store programs;

[0035] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the image fusion method of joint image registration described above.

[0036] The beneficial effects of the present application are as follows: the image fusion method of joint image registration provided in the present application, through the image registration and fusion model, comprehensively utilizes the high-resolution advantage of visible light cameras during the day and the good imaging capabilities of infrared cameras at night or in bad weather, aligns and fuses visible light images and infrared images to obtain environmental information, significantly improving the real-time performance of the algorithm, the all-weather working capability and recognition accuracy of the vehicle environment perception system.

[0037] In addition, in the field of autonomous driving, the real-time performance of the algorithm is very important. When applying image fusion technology to the vehicle environment perception system, it also faces the problem of high real-time requirements in complex and changeable driving environments. Based on this, this application breaks through the traditional inherent mode of first aligning and then fusing, adopts an end-to-end infrared and visible light image fusion method, integrates image registration algorithm and deep learning network, and designs an image registration and fusion model to realize image registration and image fusion at the same time, which greatly improves the computational efficiency of the image registration and fusion model, making the model applicable to scenarios with high real-time requirements for vehicle environment perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic flow chart of an embodiment of the image fusion method for joint image registration provided in this application;

[0039] Figure 2 For this application Figure 1 A schematic flow chart of an embodiment of step S102;

[0040] Figure 3 Schematic diagram of the feature extraction module structure provided by this application;

[0041] Figure 4 A schematic diagram of the image reconstruction module structure provided in this application;

[0042] Figure 5 This is a flow chart of an embodiment of the method for acquiring visible light images and infrared images of the present application.

[0043] Figure 6 A schematic structural diagram of an embodiment of an image fusion device for joint image registration provided by the present application;

[0044] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0046] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts may be implemented out of sequence, and steps that do not have a logical contextual relationship may be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the contents of this application, may add one or more other operations to the flowcharts or remove one or more operations from the flowcharts. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0047] The descriptions of "first" and "second" in the embodiments of this application are only used to describe the implicit purpose and should not be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, technical features defined as "first" and "second" may explicitly or implicitly include at least one of such features. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0048] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] The present application provides an image fusion method for joint image registration, which is described below.

[0050] Figure 1 This is a flow chart of an embodiment of the image fusion method for joint image registration provided by this application, as shown in FIG. Figure 1 As shown in FIG, the image fusion method of joint image registration includes:

[0051] S101. Based on a deep learning network, construct an initial image registration and fusion model, where the initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence;

[0052] S102, inputting a plurality of preset groups of visible light images and infrared images into an initial image registration and fusion model to obtain a fused image;

[0053] S103, calculating the loss between the fused image and the visible light image and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model;

[0054] S104: Perform image fusion on the visible light image and the infrared image to be measured based on the fully trained image registration and fusion model.

[0055] Compared with the existing technology, this application uses an image registration and fusion model to comprehensively utilize the high-resolution advantage of visible light cameras during the day and the good imaging capabilities of infrared cameras at night or in bad weather, and fuses visible light images and infrared images to obtain environmental information, significantly improving the all-weather working capability and recognition accuracy of the vehicle environment perception system.

[0056] In addition, in the field of autonomous driving, the real-time performance of the algorithm is very important. When applying image fusion technology to the vehicle environment perception system, it also faces the problem of high real-time requirements in complex and changeable driving environments. Based on this, this application breaks through the traditional inherent mode of first aligning and then fusing, adopts an end-to-end infrared and visible light image fusion method, integrates image registration algorithm and deep learning network, and designs an image registration and fusion model to realize image registration and image fusion at the same time, which greatly improves the computational efficiency of the image registration and fusion model, making the model applicable to scenarios with high real-time requirements for vehicle environment perception.

[0057] In order to better perform image fusion, in some embodiments, such as Figure 2 As shown, step S102 includes:

[0058] S201, inputting the visible light image and the infrared image into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features;

[0059] It should be noted that the feature extraction module is used to capture the unified feature information of multimodal information, which helps to transform the multimodal registration task into a single-modal registration task.

[0060] It should also be noted that in order to better construct an image registration fusion model for the complex environment of the vehicle, in some embodiments, the feature extraction module structure is as follows: Figure 3As shown, the feature extraction module is composed of a number of stacked convolution units, including ten convolution units connected in sequence. The first, third and ninth convolution units each include a convolution layer, a normalization layer and an excitation layer connected in sequence. The second and fourth convolution units each include a convolution layer, a normalization layer, an excitation layer and a maximum pooling layer connected in sequence. The fifth and seventh convolution units each include an upsampling layer, a convolution layer and an excitation layer connected in sequence. The sixth and eighth convolution units include a splicing layer, a convolution layer, a normalization layer and an excitation layer connected in sequence. The tenth convolution unit includes a convolution layer. The output of the first convolution unit is jump-connected to the input of the eighth convolution unit, and the output of the second convolution unit is jump-connected to the input of the sixth convolution unit. The normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

[0061] S202, performing feature point matching on initial features of the visible light image and the infrared image based on a feature matching module to obtain geometric deformation parameters, and registering the initial features based on the geometric deformation parameters to obtain registration features;

[0062] It should be noted that the feature matching module performs global correction, uses the feature information extracted from the feature to establish matching constraints, and trains the network model through the constraints to correct local parallax and output the registered feature information. In addition, after the initial matching of the feature points, the mismatched points are also removed.

[0063] S203 , performing feature fusion and feature reconstruction on the registration features based on the image reconstruction module to obtain fused features.

[0064] It should be noted that the image reconstruction module integrates the registration features to generate the final fused image.

[0065] It should also be noted that the image reconstruction module structure is as follows Figure 4 As shown in the figure, the image reconstruction module includes a 3D convolution layer, a BN layer, a ReLu layer and a maximum pooling layer connected in sequence. The spatial convolution module is used to fuse the registration feature information to obtain the final fused image. The spatial convolution can extract the characteristics of sliding on the channel dimension. A network structure based on different step length fusion is designed to enhance the model's ability to learn image features while solving the redundant information in the feature information.

[0066] In order to improve the quality of visible light images and infrared images, in some embodiments, such as Figure 5 As shown, several preset groups of visible light images and infrared images are obtained by the following method:

[0067] S501, acquiring several groups of visible light images and infrared images;

[0068] It should be noted that the steps for collecting several sets of visible light images and infrared images include: first, installing industrial sensors in an array, specifically simulating vehicle data collection, building an industrial sensor array fixture, and setting the internal and external parameters of the industrial sensors according to actual needs; then collecting a multi-source information data set for highways. In order to ensure the scale and diversity of the data set, the multi-source information data is mainly collected from road traffic data in various areas of Wuhan to construct multi-source information fusion data.

[0069] S502 : performing grayscale conversion, denoising, and correction processing on the plurality of groups of visible light images and infrared images to obtain the plurality of groups of pre-processed visible light images and infrared images.

[0070] It should be noted that the several groups of visible light images and infrared images collected are color images. Grayscale processing of the images to generate grayscale images can reduce the amount of calculation; using filtering algorithms to remove noise from grayscale images can improve image quality; finally, geometric correction, color correction and alignment of the images can eliminate image distortion caused by factors such as camera installation position and angle.

[0071] In order to better train the image registration fusion model, in some embodiments, the loss function for calculating the information loss is:

[0072]

[0073]

[0074]

[0075] Where, represents the intensity information loss function of the fused image, the visible light image, and the infrared image, Represents the gradient information loss function of the fused image, visible light image and infrared image, Indicates a trade-off and The weight parameter, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image, Indicates that the similarity between the true matrix and the estimated matrix is measured by calculating the L2 norm. Represents the mean pixel value of the matrix. Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

[0076] In order to better implement an image fusion method in an embodiment of the present application, based on an image fusion method, correspondingly, Figure 6 As shown, the embodiment of the present application further provides an image fusion device 600 for joint image registration, comprising:

[0077] A model construction unit 601 is used to construct an initial image registration and fusion model based on a deep learning network. The initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence.

[0078] A model training unit 602 is configured to input a plurality of preset sets of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, calculate the information loss between the fused image and the visible light image and the infrared image, and train the initial image registration and fusion model based on the information loss to obtain a fully trained image registration and fusion model.

[0079] The image fusion unit 603 is used to perform image fusion on the visible light image and the infrared image to be measured based on the trained image registration and fusion model.

[0080] The image fusion device 600 for joint image registration provided in the above embodiment can implement the technical solution described in the above embodiment of the image fusion method for joint image registration. The specific implementation principles of the above units can refer to the corresponding contents in the above embodiment of the image fusion method for joint image registration, which will not be repeated here.

[0081] like Figure 7 As shown, the present application also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702 and a display 703. Figure 7 Only some of the components of the electronic device 700 are shown, but it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0082] In some embodiments, the memory 702 may be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. In other embodiments, the memory 702 may also be an external storage device of the electronic device 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 700.

[0083] Furthermore, the memory 702 may include both an internal storage unit of the electronic device 700 and an external storage device. The memory 702 is used to store application software installed in the electronic device 700 and various data.

[0084] In some embodiments, the processor 701 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 702, such as the image fusion method for joint image registration in the present application.

[0085] In some embodiments, display 703 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 703 is used to display information on electronic device 700 and to display a visual user interface. Components 701-703 of electronic device 700 communicate with each other via a system bus.

[0086] In some embodiments of the present application, when the processor 701 executes the image fusion program of the joint image registration in the memory 702, the following steps may be implemented:

[0087] Based on the deep learning network, an initial image registration and fusion model is constructed. The initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence.

[0088] Inputting several preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image;

[0089] Calculate the information loss between the fused image and the visible light image and infrared image, and train the initial image registration fusion model based on the information loss to obtain a fully trained image registration fusion model;

[0090] Image fusion is performed on the visible light image and infrared image to be tested based on the well-trained image registration and fusion model.

[0091] It should be understood that, when the processor 701 executes the image fusion program of the joint image registration in the memory 702 , in addition to the above functions, it can also implement other functions. For details, please refer to the description of the corresponding method embodiment above.

[0092] Furthermore, the embodiment of the present application does not specifically limit the type of the electronic device 700 mentioned. The electronic device 700 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, or the like. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft, or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present application, the electronic device 700 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0093] The above is a detailed introduction to the image fusion method of joint image registration provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0094] The above is only a preferred specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in this application should be covered by the scope of protection of the present application.

Claims

1. An image fusion method for joint image registration, characterized in that: include: Based on a deep learning network, an initial image registration fusion model is constructed, wherein the initial image registration fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence, wherein the feature extraction module is composed of a plurality of convolutional units stacked together, including ten convolutional units connected in sequence, wherein the first, third, and ninth convolutional units each include a convolutional layer, a normalization layer, and an excitation layer connected in sequence, the second and fourth convolutional units each include a convolutional layer, a normalization layer, an excitation layer, and a maximum pooling layer connected in sequence, the fifth and seventh convolutional units each include an upsampling layer, a convolutional layer, and an excitation layer connected in sequence, the sixth and eighth convolutional units include a splicing layer, a convolutional layer, a normalization layer, and an excitation layer connected in sequence, the tenth convolutional unit includes a convolutional layer, the output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit; Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image; Calculating the loss between the fused image and the visible light image and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model; Image fusion is performed on the visible light image and the infrared image to be measured based on the fully trained image registration and fusion model.

2. The image fusion method of joint image registration according to claim 1, characterized in that: Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image includes: Inputting the visible light image and the infrared image into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features; Performing feature point matching on initial features of the visible light image and the infrared image based on the feature matching module to obtain geometric deformation parameters, and registering the initial features based on the geometric deformation parameters to obtain registration features; Based on the image reconstruction module, feature fusion and feature reconstruction are performed on the registration features to obtain fusion features.

3. The image fusion method of joint image registration according to claim 1, characterized in that: The normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

4. The image fusion method of joint image registration according to claim 1, characterized in that: The loss function for calculating the loss is: Where, represents the intensity information loss function of the fused image, the visible light image, and the infrared image, Represents the gradient information loss function of the fused image, visible light image and infrared image, Indicates a trade-off and The weight parameter of .

5. The image fusion method of joint image registration according to claim 4, characterized in that: The intensity information loss function and the gradient information loss function are specifically: Where, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image, Indicates that the similarity between the true matrix and the estimated matrix is measured by calculating the L2 norm. represents the average pixel value of the matrix, Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

6. The image fusion method of joint image registration according to claim 1, characterized in that: The preset sets of visible light images and infrared images are obtained by the following method: Acquire several sets of visible light images and infrared images; Grayscale conversion, denoising and correction are performed on the groups of visible light images and infrared images to obtain the pre-processed groups of visible light images and infrared images.

7. An image fusion device for joint image registration, characterized in that: include: A model construction unit is used to construct an initial image registration and fusion model based on a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence, wherein the feature extraction module is composed of a plurality of convolutional units stacked together, including ten convolutional units connected in sequence, wherein the first, third and ninth convolutional units each include a convolutional layer, a normalization layer and an excitation layer connected in sequence, the second and fourth convolutional units each include a convolutional layer, a normalization layer, an excitation layer and a maximum pooling layer connected in sequence, the fifth and seventh convolutional units each include an upsampling layer, a convolutional layer and an excitation layer connected in sequence, the sixth and eighth convolutional units include a splicing layer, a convolutional layer, a normalization layer and an excitation layer connected in sequence, the tenth convolutional unit includes a convolutional layer, the output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit; a model training unit, configured to input a plurality of preset sets of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, calculate information loss between the fused image and the visible light image and the infrared image, and train the initial image registration and fusion model based on the information loss to obtain a fully trained image registration and fusion model; The image fusion unit is used to fuse the visible light image and the infrared image to be measured based on the trained image registration and fusion model.

8. An electronic device, characterized in that: including memory and processor; The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the image fusion method for joint image registration as described in any one of claims 1 to 6.

Images