Image registration-combined image fusion method and device and electronic medium

Abstract

The invention relates to the technical field of automatic driving, in particular to an image fusion method and device combined with image registration and electronic equipment, and belongs to the technical field of automatic driving. According to the method, an image registration fusion model is constructed by using a deep learning network, and unregistered infrared / visible light images are registered and fused, so that the rapid perception of the vehicle environment is realized. The model firstly performs feature extraction on an image, then performs feature matching and calibration on extracted feature information, and finally obtains a fused image through a reconstruction module. In the network training stage, the information loss of the fusion image and the infrared / visible light image is calculated, and the model is trained based on the loss to obtain a completely trained network model. According to the method, the end-to-end network model is fused through image registration, the advantages of visible light and infrared images are comprehensively utilized to quickly obtain environment information, and the model can be applied to a vehicle environment perception high-real-time requirement scene.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an image fusion method, device and electronic medium for joint image registration. Background Art

[0002] In recent years, image fusion technology, as a major branch of multi-sensor information fusion technology, has received widespread attention. Image fusion technology integrates image information from different sensors through algorithms to generate images that are consistent with human visual perception and rich in image information. It can provide more complete image information for subsequent engineering systems, assist engineering systems in making more accurate decisions, and is one of the indispensable components in engineering system decision-making.

[0003] In the existing image fusion methods, usually only the image fusion algorithm itself is studied, assuming that the source images have been pixel-level registered. If the images that are not accurately registered are directly fused, ghosting, blurring and other phenomena will appear in the fusion results. Therefore, the existing image fusion methods have largely ignored the importance of image registration. In actual engineering applications, a semi-automatic operation mode with manual intervention is usually used to achieve image registration, and then the registered images are fused, which greatly reduces the efficiency of the image processing task. In addition, there are differences in imaging mechanisms between cross-modal images, and it is difficult to capture feature information for registration. Therefore, general image registration algorithms are difficult to apply to multimodal images with feature differences. Therefore, how to solve the precise registration of multimodal image fusion in engineering applications is an important challenge for multimodal image fusion technology.

[0004] With the rapid development of intelligent transportation systems, vehicle environment perception technology has become a core component of autonomous driving and advanced driver assistance systems. This technology integrates multiple sensors (such as radar, lidar, cameras, etc.) to obtain real-time information about the vehicle's surroundings, thereby realizing functions such as path planning, obstacle detection, and collision warning.

[0005] In the existing technology, vehicle environment perception technology mainly uses visible light vision sensors (Tesla) to obtain information about the vehicle's surrounding environment in real time. However, a single visible light vision sensor is easily interfered by external environmental factors, and the environmental information is not accurately obtained, resulting in poor recognition accuracy of the vehicle environment perception system. Combined with infrared visible light image fusion technology, it is possible to simultaneously obtain the vehicle's environmental information and thermal radiation information, enhancing the vehicle's environmental perception capabilities. Summary of the invention

[0006] In view of this, it is necessary to provide an image fusion method, device and electronic device for joint image registration. This is to solve the problem that the existing technology cannot be directly applied to applications in high real-time demand scenarios. While ensuring the real-time demand, it solves the problem that the single visible light vision sensor in the vehicle environment perception system is easily disturbed by external environmental factors, resulting in poor recognition accuracy.

[0007] In order to solve the above problems, the present application provides an image fusion method for joint image registration, comprising: An initial image registration and fusion model is constructed using a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence; Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image; Calculating the loss of the fused image, the visible light image and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model; Image fusion is performed on the visible light image and infrared image to be measured based on the fully trained image registration and fusion model.

[0008] In some possible implementations, inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image includes: The visible light image and the infrared image are input into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features; Based on the feature matching module, feature point matching is performed on the initial features of the visible light image and the infrared image to obtain geometric deformation parameters, and based on the geometric deformation parameters, the initial features are registered to obtain registration features; Based on the image reconstruction module, feature fusion and feature reconstruction are performed on the registration features to obtain fusion features.

[0009] In some possible implementations, the feature extraction module is formed by stacking a number of convolutional units.

[0010] In some possible implementations, the feature extraction module includes ten convolutional units connected in sequence, wherein the first, third and ninth convolutional units each include a convolutional layer, a normalization layer and an excitation layer connected in sequence, the second and fourth convolutional units each include a convolutional layer, a normalization layer, an excitation layer and a maximum pooling layer connected in sequence, the fifth and seventh convolutional units each include an upsampling layer, a convolutional layer and an excitation layer connected in sequence, the sixth and eighth convolutional units include a splicing layer, a convolutional layer, a normalization layer and an excitation layer connected in sequence, the tenth convolutional unit includes a convolutional layer, the output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit.

[0011] In some possible implementations, the normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

[0012] In some possible implementations, the loss function used when training the initial image registration fusion model is:

[0013] In the formula, represents the intensity information loss function of the fused image, the visible light image and the infrared image, Represents the gradient information loss function of the fused image, the visible light image, and the infrared image, Indicates a trade-off and The weight parameter of .

[0014] In some possible implementations, the intensity information loss function and the gradient information loss function are specifically:

[0015]

[0016] In the formula, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image. It means that the similarity between the true matrix and the estimated matrix is ​​measured by calculating the L2 norm. Represents the mean pixel value of the matrix. Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

[0017] In some possible implementations, the preset groups of visible light images and infrared images are obtained by the following method: Acquire several groups of visible light images and infrared images; Grayscale, denoising and correction processing are performed on the several groups of visible light images and infrared images to obtain several groups of pre-processed visible light images and infrared images.

[0018] The present application also provides an image fusion device for joint image registration, comprising: A model building unit, used to build an initial image registration and fusion model based on a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence; A model training unit is used to input a preset number of groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, calculate the information loss between the fused image and the visible light image and the infrared image, and train the initial image registration and fusion model based on the information loss to obtain a fully trained image registration and fusion model; The image fusion unit is used to fuse the visible light image and the infrared image to be measured based on the trained image registration and fusion model.

[0019] The present application also provides an electronic device, including a memory and a processor; The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the image fusion method of joint image registration described above.

[0020] The beneficial effects of the present application are as follows: the image fusion method of joint image registration provided in the present application, through the image registration and fusion model, comprehensively utilizes the high-resolution advantage of visible light cameras during the day and the good imaging capabilities of infrared cameras at night or in bad weather, to register and fuse visible light images and infrared images to obtain environmental information, thereby significantly improving the real-time performance of the algorithm, the all-weather working capability and recognition accuracy of the vehicle environmental perception system.

[0021] In addition, in the field of autonomous driving, the real-time performance of the algorithm is very important. When applying image fusion technology to the vehicle environment perception system, it also faces the problem of high real-time requirements in complex and changeable driving environments. Based on this, this application breaks through the traditional inherent mode of first registration and then fusion, adopts an end-to-end infrared and visible light image fusion method, integrates image registration algorithm and deep learning network, and designs an image registration and fusion model that simultaneously realizes image registration and image fusion, which greatly improves the computational efficiency of the image registration and fusion model, so that the model can be applied to scenarios with high real-time requirements for vehicle environment perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A schematic diagram of an embodiment of the image fusion method for joint image registration provided in the present application; Figure 2 For this application Figure 1 A schematic flow chart of an embodiment of step S102; Figure 3 A schematic diagram of the feature extraction module structure provided for this application; Figure 4 A schematic diagram of the image reconstruction module structure provided in this application; Figure 5 The figure is a flow chart of an embodiment of the method for acquiring visible light images and infrared images of the present application.

[0023] Figure 6 A schematic diagram of the structure of an embodiment of an image fusion device for joint image registration provided by the present application; Figure 7 A schematic diagram of the structure of an embodiment of an electronic device provided in this application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowcharts, or may remove one or more operations from the flowcharts, under the guidance of the content of this application. Some of the block diagrams shown in the accompanying drawings are functional entities, and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0026] The descriptions of "first", "second", etc. involved in the embodiments of the present application are only used to describe the implicit purpose, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" and "second" may explicitly or implicitly include at least one of the features. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist, for example: A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The present application provides an image fusion method for joint image registration, which is described below respectively.

[0029] Figure 1 A schematic flow chart of an embodiment of the image fusion method for joint image registration provided in this application is as follows: Figure 1 As shown, the image fusion method of joint image registration includes: S101. Based on a deep learning network, construct an initial image registration and fusion model, where the initial image registration and fusion model includes a feature extraction module, a feature matching module, and an image reconstruction module connected in sequence; S102, inputting a preset number of groups of visible light images and infrared images into an initial image registration and fusion model to obtain a fused image; S103, calculating the loss of the fused image, the visible light image, and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model; S104, performing image fusion on the visible light image and the infrared image to be measured based on the fully trained image registration and fusion model.

[0030] Compared with the existing technology, this application uses an image registration and fusion model to comprehensively utilize the high-resolution advantages of visible light cameras during the day and the good imaging capabilities of infrared cameras at night or in bad weather, and fuses visible light images and infrared images to obtain environmental information, significantly improving the all-weather working capability and recognition accuracy of the vehicle environment perception system.

[0031] In addition, in the field of autonomous driving, the real-time performance of the algorithm is very important. When applying image fusion technology to the vehicle environment perception system, it also faces the problem of high real-time requirements in complex and changeable driving environments. Based on this, this application breaks through the traditional inherent mode of first registration and then fusion, adopts an end-to-end infrared and visible light image fusion method, integrates image registration algorithm and deep learning network, and designs an image registration and fusion model that simultaneously realizes image registration and image fusion, which greatly improves the computational efficiency of the image registration and fusion model, so that the model can be applied to scenarios with high real-time requirements for vehicle environment perception.

[0032] In order to better perform image fusion, in some embodiments, such as Figure 2 As shown, step S102 includes: S201, inputting the visible light image and the infrared image into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features; It should be noted that the feature extraction module is used to capture the unified feature information of multimodal information, which helps to transform the multimodal registration task into a single-mode registration.

[0033] It should also be noted that in order to better construct an image registration fusion model for complex vehicle environments, in some embodiments, the feature extraction module structure is as follows: Figure 3 As shown, the feature extraction module is composed of a number of convolutional units stacked together, including ten convolutional units connected in sequence. The first, third and ninth convolutional units all include convolutional layers, normalization layers and excitation layers connected in sequence. The second and fourth convolutional units all include convolutional layers, normalization layers, excitation layers and maximum pooling layers connected in sequence. The fifth and seventh convolutional units all include upsampling layers, convolutional layers and excitation layers connected in sequence. The sixth and eighth convolutional units include splicing layers, convolutional layers, normalization layers and excitation layers connected in sequence. The tenth convolutional unit includes a convolutional layer. The output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit. The normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

[0034] S202, performing feature point matching on initial features of the visible light image and the infrared image based on a feature matching module to obtain geometric deformation parameters, and registering the initial features based on the geometric deformation parameters to obtain registration features; It should be noted that the feature matching module performs global correction, uses the feature information extracted from the feature to establish matching constraints, trains the network model through constraints, corrects local parallax, and outputs the registered feature information. In addition, after the initial matching of the feature points, the mismatched points are also removed.

[0035] S203: Based on the image reconstruction module, feature fusion and feature reconstruction are performed on the registration features to obtain fusion features.

[0036] It should be noted that the image reconstruction module integrates the registration features to generate the final fused image.

[0037] It should also be noted that the image reconstruction module structure is as follows Figure 4 As shown in the figure, the image reconstruction module includes a 3D convolution layer, a BN layer, a ReLu layer and a maximum pooling layer connected in sequence. The spatial convolution module is used to fuse the registration feature information to obtain the final fused image. The spatial convolution can extract the characteristics of sliding on the channel dimension. A network structure based on different step length fusion is designed to enhance the model's ability to learn image features while solving the redundant information in the feature information.

[0038] In order to improve the quality of visible light images and infrared images, in some embodiments, such as Figure 5 As shown, several preset groups of visible light images and infrared images are obtained by the following method: S501, acquiring several groups of visible light images and infrared images; It should be noted that the steps for collecting several groups of visible light images and infrared images include: first installing industrial sensors in an array, specifically simulating vehicle data collection, building an industrial sensor array fixture, and setting the internal and external parameters of the industrial sensors according to actual needs; then collecting a highway multi-source information data set. In order to ensure the scale and diversity of the data set, the multi-source information data is mainly collected from road traffic data in various areas of Wuhan to construct multi-source information fusion data.

[0039] S502 , gray-scaling, denoising and correcting are performed on a plurality of groups of visible light images and infrared images to obtain a plurality of groups of pre-processed visible light images and infrared images.

[0040] It should be noted that the several groups of visible light images and infrared images collected are color images. Grayscale processing is performed on the images to generate grayscale images, which can reduce the amount of calculation; the filtering algorithm is used to remove noise from the grayscale images to improve the image quality; finally, the images are geometrically corrected, color corrected and aligned to eliminate image distortion caused by factors such as the camera installation position and angle.

[0041] In order to better train the image registration fusion model, in some embodiments, the loss function for calculating the information loss is:

[0042]

[0043]

[0044] In the formula, represents the intensity information loss function of the fused image, the visible light image and the infrared image, Represents the gradient information loss function of the fused image, the visible light image, and the infrared image, Indicates a trade-off and The weight parameter, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image. It means that the similarity between the true matrix and the estimated matrix is ​​measured by calculating the L2 norm. Represents the mean pixel value of the matrix. Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

[0045] In order to better implement an image fusion method in an embodiment of the present application, based on an image fusion method, correspondingly, Figure 6 As shown, the embodiment of the present application further provides an image fusion device 600 for joint image registration, comprising: A model building unit 601 is used to build an initial image registration and fusion model based on a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence; The model training unit 602 is used to input a preset number of groups of visible light images and infrared images into the initial image registration fusion model to obtain a fused image, calculate the information loss between the fused image and the visible light image and the infrared image, and train the initial image registration fusion model based on the information loss to obtain a fully trained image registration fusion model; The image fusion unit 603 is used to fuse the visible light image and the infrared image to be measured based on the well-trained image registration and fusion model.

[0046] The image fusion device 600 for joint image registration provided in the above embodiment can implement the technical solution described in the above embodiment of the image fusion method for joint image registration. The specific implementation principles of the above units can refer to the corresponding contents in the above embodiment of the image fusion method for joint image registration, which will not be repeated here.

[0047] like Figure 7 As shown, the present application also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702 and a display 703. Figure 7Only some components of the electronic device 700 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0048] In some embodiments, the memory 702 may be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. In other embodiments, the memory 702 may also be an external storage device of the electronic device 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 700.

[0049] Furthermore, the memory 702 may include both an internal storage unit of the electronic device 700 and an external storage device. The memory 702 is used to store application software installed in the electronic device 700 and various data.

[0050] In some embodiments, the processor 701 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 702, such as the image fusion method of joint image registration in the present application.

[0051] In some embodiments, the display 703 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 703 is used to display information on the electronic device 700 and to display a visual user interface. The components 701-703 of the electronic device 700 communicate with each other via a system bus.

[0052] In some embodiments of the present application, when the processor 701 executes the image fusion program of the joint image registration in the memory 702, the following steps may be implemented: Based on the deep learning network, an initial image registration and fusion model is constructed. The initial image registration and fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence. Inputting a preset number of groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image; Calculate the information loss of the fused image, the visible light image and the infrared image, train the initial image registration fusion model based on the information loss, and obtain a fully trained image registration fusion model; Image fusion is performed on the visible light image and infrared image to be tested based on the fully trained image registration and fusion model.

[0053] It should be understood that: when the processor 701 executes the image fusion program of joint image registration in the memory 702, in addition to the above functions, other functions may also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0054] Furthermore, the embodiment of the present application does not specifically limit the type of the electronic device 700 mentioned, and the electronic device 700 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present application, the electronic device 700 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0055] The above is a detailed introduction to the image fusion method for joint image registration provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0056] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. An image fusion method for joint image registration, characterized in that: include: Based on the deep learning network, an initial image registration fusion model is constructed, wherein the initial image registration fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence; Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image; Calculating the loss of the fused image, the visible light image and the infrared image, and training the initial image registration fusion model based on the loss to obtain a fully trained image registration fusion model; Image fusion is performed on the visible light image and infrared image to be measured based on the fully trained image registration and fusion model.

2. The image fusion method for joint image registration according to claim 1, characterized in that: Inputting a plurality of preset groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, including: The visible light image and the infrared image are input into the initial image registration and fusion model, and the feature extraction module extracts features from the visible light image and the infrared image respectively to obtain initial features; Based on the feature matching module, feature point matching is performed on the initial features of the visible light image and the infrared image to obtain geometric deformation parameters, and based on the geometric deformation parameters, the initial features are registered to obtain registration features; Based on the image reconstruction module, feature fusion and feature reconstruction are performed on the registration features to obtain fusion features.

3. The image fusion method of joint image registration according to claim 1, characterized in that: The feature extraction module is composed of a plurality of stacked convolution units.

4. The image fusion method for joint image registration according to claim 3, characterized in that: The feature extraction module includes ten convolutional units connected in sequence, wherein the first, third and ninth convolutional units each include a convolutional layer, a normalization layer and an excitation layer connected in sequence, the second and fourth convolutional units each include a convolutional layer, a normalization layer, an excitation layer and a maximum pooling layer connected in sequence, the fifth and seventh convolutional units each include an upsampling layer, a convolutional layer and an excitation layer connected in sequence, the sixth and eighth convolutional units include a splicing layer, a convolutional layer, a normalization layer and an excitation layer connected in sequence, the tenth convolutional unit includes a convolutional layer, the output of the first convolutional unit is jump-connected to the input of the eighth convolutional unit, and the output of the second convolutional unit is jump-connected to the input of the sixth convolutional unit.

5. The image fusion method for joint image registration according to claim 4, characterized in that: The normalization layer is an Instance Normalization layer, and the excitation layer is an LReLU.

6. The image fusion method of joint image registration according to claim 1, characterized in that: The loss function for calculating the information loss is: In the formula, represents the intensity information loss function of the fused image, the visible light image and the infrared image, Represents the gradient information loss function of the fused image, the visible light image, and the infrared image, Indicates a trade-off and The weight parameter of .

7. The image fusion method of joint image registration according to claim 6, characterized in that: The intensity information loss function and the gradient information loss function are specifically: In the formula, represents the fused image, represents a visible light image, represents an infrared image, and Indicates the size of the image. It means that the similarity between the true matrix and the estimated matrix is ​​measured by calculating the L2 norm. Represents the mean pixel value of the matrix. Represents the gradient information of the image, Indicates obtaining the maximum gradient information from visible light images and infrared images.

8. The image fusion method of joint image registration according to claim 1, characterized in that: The preset groups of visible light images and infrared images are obtained by the following method: Acquire several groups of visible light images and infrared images; Grayscale, denoising and correction processing are performed on the several groups of visible light images and infrared images to obtain several groups of pre-processed visible light images and infrared images.

9. An image fusion device for joint image registration, characterized in that: include: A model building unit, used to build an initial image registration and fusion model based on a deep learning network, wherein the initial image registration and fusion model includes a feature extraction module, a feature matching module and an image reconstruction module connected in sequence; A model training unit is used to input a preset number of groups of visible light images and infrared images into the initial image registration and fusion model to obtain a fused image, calculate the information loss between the fused image and the visible light image and the infrared image, and train the initial image registration and fusion model based on the information loss to obtain a fully trained image registration and fusion model; The image fusion unit is used to fuse the visible light image and the infrared image to be measured based on the trained image registration and fusion model.

10. An electronic device, characterized in that: including memory and processor; The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the image fusion method for joint image registration as described in any one of claims 1 to 8 above.

Images