Fusion method of visible light image and infrared image

Abstract

The invention discloses a fusion method of a visible light image and an infrared image. The method comprises the following steps: acquiring a target visible light image and a target infrared image, and inputting the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model comprises a first fusion network corresponding to the fusion image of the first model and a second fusion network corresponding to the fusion image of the second model; a third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superposed to obtain a target fusion image, so that key information in the fusion image is enhanced; and the target can be accurately identified and positioned under severe conditions of high illumination, smoke and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for fusing a visible light image and an infrared image. Background Art

[0002] Image fusion technology is an important branch of data fusion. It uses multiple image information of the same scene to output a fused image that is more suitable for visual perception. Among them, visible light images can reflect the details of the scene under low light intensity, but overexposure under high light intensity causes the loss of details in the bright area of ​​the image. The grayscale of infrared images is determined by the temperature difference between the target and the scene, and they cannot show the natural color of the scene. However, there is rich infrared information under high light intensity, so the fusion of visible light images and infrared images can improve the clarity and information content of the fused image.

[0003] In related technologies, the fusion of visible light images and infrared images includes pixel level, feature level and model level. The pixel level fusion method is simple and intuitive, but may lead to information loss; the feature level fusion method can better retain details and features, but has high requirements for the accuracy of feature extraction; the model level fusion method obtains the best results through modeling and optimization, and has high requirements for model selection and parameter adjustment. Summary of the invention

[0004] The present invention provides a method for fusing a visible light image and an infrared image, so as to fuse the details of the visible light image with the thermal radiation information of the infrared image and generate an image with higher quality.

[0005] According to one aspect of the present invention, a method for fusing a visible light image and an infrared image is provided, the method comprising:

[0006] Acquire a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image;

[0007] A third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image.

[0008] According to another aspect of the present invention, a device for fusing a visible light image and an infrared image is provided, which may include:

[0009] A first image fusion module is used to obtain a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image;

[0010] The second image fusion module is used to determine a third model fusion image according to the first model fusion image and the second model fusion image, and superimpose the first model fusion image and the third model fusion image to obtain a target fusion image.

[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for fusing visible light images and infrared images described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for fusing visible light images and infrared images described in any embodiment of the present invention when executed.

[0016] The technical solution of the embodiment of the present invention is as follows: first, a target visible light image and a target infrared image are acquired, and the target visible light image and the target infrared image are input into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image. The image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image. The first model fusion image and the second model fusion image are generated by two different fusion networks, respectively, which will highlight the details and features of the visible light image and the infrared image, thereby providing a more diverse image information display method and effectively enhancing the image quality and information content; then, a third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image. By generating and superimposing multiple fusion images, the feature types and information amount of the target fusion image can be increased, and the visual effect of the overall image can be improved. The technical solution of the embodiment of the present invention generates a first model fusion image and a second model fusion image respectively through two independent fusion networks, then determines a third model fusion image, and then superimposes the first and third model fusion images to obtain a target fusion image, which solves the problems of information loss of visible light images and infrared images, high feature accuracy requirements and complex parameter adjustment in related technologies, enhances the key information in the fused image, and is conducive to accurately identifying and positioning targets under harsh conditions such as high light and smoke.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a flow chart of a method for fusing a visible light image and an infrared image provided according to the first embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for fusing a visible light image and an infrared image provided according to Embodiment 2 of the present invention;

[0021] Figure 3 is a flow chart of another visible light image and infrared image fusion method provided according to the second embodiment of the present invention;

[0022] Figure 4 is a flow chart of a method for fusing a visible light image and an infrared image provided according to Embodiment 3 of the present invention;

[0023] Figure 5 is a schematic diagram of the structure of a visible light image and infrared image fusion device provided according to Embodiment 4 of the present invention;

[0024] Figure 6 It is a structural schematic diagram of an electronic device for implementing the method for fusing visible light images and infrared images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0031] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a method for fusing a visible light image and an infrared image is provided for Embodiment 1 of the present invention. This embodiment is applicable to the case where the details of a visible light image are fused with the thermal radiation information of an infrared image to generate an image of higher quality. The method can be executed by a device for fusing a visible light image and an infrared image. The device for fusing a visible light image and an infrared image can be implemented in the form of hardware and / or software, and optionally, can be implemented by an electronic device, which can be a mobile terminal, a PC, a server, etc.

[0036] like Figure 1 As shown, the method may specifically include:

[0037] S110. Acquire a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image.

[0038] Among them, the target visible light image can be understood as an image within the visible spectrum, which is usually an image that can be directly seen by the eyes and can be used to perceive color and brightness information. It can be understood that since the visible light image is sensitive to the ambient lighting conditions, it may not be able to capture the object clearly when the light intensity is low; while in the case of strong light, the object will be overexposed, resulting in loss of details in the bright area of ​​the image, whitening of the color, and reduction of the overall visual effect. The target infrared image can be understood as an image taken with infrared rays, which can capture information such as thermal radiation emitted by the object, is independent of external lighting conditions, and can identify objects under high light or smoke conditions. The image fusion model can be understood as a pre-trained deep learning model, which is used to output a first model fusion image and a second model fusion image after inputting the target visible light image and the target infrared image. Exemplarily, the image fusion model may include a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image. The first model fusion image can be understood as an image generated by the first fusion network in the image fusion model, which is the result of integrating the visible light image and the infrared image information, and is used to extract one type of information feature. The second model fusion image can be understood as an image generated by the second fusion network in the image fusion model, which also fuses the information of the visible light image and the infrared image, but compared with the first model fusion image, it is used to extract different types of features to provide supplementary information. The first fusion network can be understood as a sub-model in the image fusion model, which is used to generate the first model fusion image. Correspondingly, the second fusion network can be understood as another sub-model in the image fusion model, which is used to generate the second model fusion image. The first fusion network and the second fusion network fuse the target visible light image and the target infrared image in different ways.

[0039] S120: Determine a third model fusion image according to the first model fusion image and the second model fusion image, and superimpose the first model fusion image and the third model fusion image to obtain a target fusion image.

[0040] The third model fusion image can be understood as a new image obtained by further processing the first model fusion image and the second model fusion image, which is used to optimize and integrate information and provide more comprehensive image information. The target fusion image can be understood as a fusion image formed by superimposing the first model fusion image and the third model fusion image. The target fusion image integrates the image information of the target visible light image and the target infrared image, and is used to provide richer and higher quality image information.

[0041] On the basis of the above solution, determining a third model fusion image according to the first model fusion image and the second model fusion image based on the following formula includes:

[0042] I xor = rescale(I fin +I feg -2I fin I feg ),

[0043] Among them, I xor represents the third model fusion image, I fin represents the first model fusion image, I feg Represents the second model fusion image, and rescale(.) is a function that scales the input image to [0,1].

[0044] In this embodiment, it is necessary to first calculate the first model fusion image I fin Fusion image I with the second model feg The pixel-level product of fin +I feg -2I fin I feg Finally, the normalization function rescale(·) is used to scale the result to obtain the third model fusion image I x9r The normalization function rescale(·) is used to ensure that the image features of the target visible light image and the target infrared image can be effectively integrated during the image fusion process, while avoiding problems such as image data loss and overexposure.

[0045] In the above method, by combining the first model fusion image I fin Fusion image I with the second model feg The features of the target images are taken into consideration, and the similarities and differences between them are taken into consideration, and finally a third model fusion image is generated, which helps to improve the information content and reliability of the target fusion image.

[0046] The technical solution of the embodiment of the present invention is as follows: first, a target visible light image and a target infrared image are acquired, and the target visible light image and the target infrared image are input into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image, wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image, and the first model fusion image and the second model fusion image are respectively generated by two different fusion networks, which will highlight different details and features, thereby providing more diverse information display methods and effectively enhancing the quality and information content of the image; then, a third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image. By generating and superimposing multiple fusion images, the feature types and information amount of the target fusion image can be increased, and the visual effect of the overall image can be improved. The technical solution of the embodiment of the present invention generates a first model fusion image and a second model fusion image respectively through two independent fusion networks, then determines a third model fusion image, and then superimposes the first and third model fusion images to obtain a target fusion image, which solves the problems of information loss of visible light images and infrared images, high feature accuracy requirements and complex parameter adjustment in related technologies, enhances the key information in the fused image, and is conducive to accurately identifying and positioning targets under harsh conditions such as high light and smoke.

[0047] Embodiment 2

[0048] Figure 2A flowchart of a method for fusing a visible light image and an infrared image provided in Embodiment 2 of the present invention, this embodiment can be combined with the above embodiments. The model parameters are optimized through iterative training to improve the quality and effect of the fusion of visible light and infrared images. Optionally, before the target visible light image and the target infrared image are input into the pre-trained image fusion model, the method further includes: obtaining a sample visible light image and a sample infrared image, inputting the sample visible light image and the sample infrared image into the model to be trained to obtain a first sample fusion image and a second sample fusion image, and determining a third sample fusion image according to the first sample fusion image and the second sample fusion image; respectively extracting the first target feature corresponding to the third sample fusion image, the second target feature corresponding to the sample visible light image, and the third target feature corresponding to the sample infrared image through at least one target feature extraction network, and determining a first loss according to the first target feature, the second target feature, and the third target feature, wherein the target feature extraction network includes a texture feature extraction network and / or a semantic feature extraction network; determining a target loss according to the first loss, and adjusting the parameters of the model to be trained according to the target loss to obtain an image fusion model. For specific implementation, please refer to the description of this embodiment. Among them, the technical features that are the same or similar to the above-mentioned embodiments are not repeated here.

[0049] like Figure 2 As shown, the method may specifically include:

[0050] S210. Acquire a sample visible light image and a sample infrared image, input the sample visible light image and the sample infrared image into a model to be trained to obtain a first sample fusion image and a second sample fusion image, and determine a third sample fusion image according to the first sample fusion image and the second sample fusion image.

[0051] Among them, the sample visible light image can be understood as an image obtained by a camera. The sample visible light image can reproduce the color and details of the scene, but its performance is limited in low-light or obscured environments. In high-light environments, objects in the image are prone to overexposure, resulting in loss of details in bright areas and whitening of colors, thereby reducing the overall visual quality. The sample infrared image can be understood as captured by an infrared camera, mainly recording the infrared radiation emitted by the object, which can show the heat distribution, but the presentation of texture and details is poor. The sample visible light image and the sample infrared image are used to train the model to be trained. The model to be trained can be understood as a model for learning and optimizing the image fusion process. Exemplarily, the model to be trained can be a model based on deep learning, which gradually improves its fusion effect by inputting a large number of sample images and adjusting parameters through repeated iterations. The first sample fused image and the second sample fused image can be understood as preliminary fusion results generated based on the input sample visible light image and sample infrared image, respectively focusing on different aspects of information features. The second sample fused image can have a corresponding label. The label can be understood as an image that combines the visible light image and the infrared image to compensate for the actual information of the image, and integrates the texture features of the visible light image and the semantic features of the infrared image. Specifically, the second sample fused image can be obtained by multiplying the sample visible light image and the sample infrared image. The third sample fused image can be understood as an image obtained based on the first sample fused image and the second sample fused image. The third sample fused image is intended to integrate the image information of the first sample fused image and the second sample fused image to provide more comprehensive and accurate visual information.

[0052] S220. Extract, through at least one target feature extraction network, a first target feature corresponding to the third sample fusion image, a second target feature corresponding to the sample visible light image, and a third target feature corresponding to the sample infrared image, respectively, and determine a first loss according to the first target feature, the second target feature, and the third target feature.

[0053] The target feature extraction network can be used to extract key features of the image content from the input image, which may include but is not limited to convolutional layers and pooling layers. The target feature extraction network may include a texture feature extraction network and / or a semantic feature extraction network. The texture feature extraction network may be a network for extracting texture features in an image, such as texture patterns and edges in an image. The semantic feature extraction network may be a network for extracting semantic features in an image, such as object categories and scene content.

[0054] The first target feature can be understood as a target feature extracted from the third sample fused image, including texture features and semantic features of the sample visible light image and the sample infrared image. The second target feature can be understood as a target feature extracted from the sample visible light image, and can include semantic features extracted from the sample visible light image, such as the subject contained in the image and the category corresponding to the subject. The third target feature can be understood as a target feature extracted from the sample infrared image, and can include attributes such as color, brightness and texture of the object. The first loss can be a loss value calculated from the extracted first target feature, the second target feature and the third target feature, which is used to indicate the degree of difference between the third sample fused image and the sample visible light image and the sample infrared image.

[0055] Specifically, by minimizing the first loss, the parameters of the model to be trained can be adjusted to improve the fusion capability of the model to be trained, so that the fused image retains key image information in the sample visible light image and the sample infrared image.

[0056] On the basis of the above scheme, optionally, the first target feature corresponding to the third sample fusion image, the second target feature corresponding to the sample visible light image, and the third target feature corresponding to the sample infrared image are respectively extracted through at least one target feature extraction network, including: inputting the third sample fusion image into a target feature extraction model to obtain a first target feature corresponding to the third sample fusion image; inputting an enhanced image of the sample visible light image into the target feature extraction model to obtain a second target feature corresponding to the sample visible light image; and inputting the sample infrared image into the target feature extraction model to obtain a third target feature corresponding to the sample infrared image.

[0057] The target feature extraction model is used to extract key features from the image, including but not limited to texture features and semantic features. The enhanced image of the sample visible light image can be understood as an image used to reflect the properties of the enhanced visible light image. Exemplarily, the sample visible light image can be enhanced by grayscale histogram normalization to obtain the enhanced image of the sample visible light image. The grayscale histogram normalization is used to readjust the grayscale value distribution of the sample visible light image so that the sample visible light image spans a larger grayscale range, thereby enhancing the lighting conditions and exposure level of the sample visible light image.

[0058] S230. Determine a target loss according to the first loss, and adjust parameters of the model to be trained according to the target loss to obtain an image fusion model.

[0059] The target loss can be understood as the overall loss value calculated based on the first loss, which is used to indicate the adjustment of the parameters of the model to be trained to obtain the image fusion model. The image fusion model is different from the model to be trained and can be understood as a trained model that can fuse the visible light image and the infrared image to generate a high-quality fused image.

[0060] Based on the above solution, optionally, after obtaining the first sample fusion image and the second sample fusion image, the method further includes: determining the second loss of the model to be trained according to the sample visible light image, the sample infrared image, the first sample fusion image and the second sample fusion image. Further, determining the target loss according to the first loss may include: determining the target loss according to the first loss and the second loss.

[0061] The second loss can be understood as a loss value calculated according to the sample visible light image, the sample infrared image, the first sample fusion image and the second sample fusion image.

[0062] On the basis of the above scheme, optionally, determining the second loss of the model to be trained according to the sample visible light image, the sample infrared image, the first sample fusion image and the second sample fusion image includes: determining a first brightness image according to the sample visible light image and the sample infrared image, and determining a third sub-loss between the first sample fusion image and the first brightness image; determining a second brightness image according to a gradient image of the sample visible light image and a gradient image of the sample infrared image, and determining a fourth sub-loss between the gradient image of the first sample fusion image and the second brightness image; multiplying the sample visible light image and the sample infrared image to obtain a third sample fusion image, determining a fifth sub-loss between the second sample fusion image and the third sample fusion image, and determining the second loss of the model to be trained according to the third sub-loss, the fourth sub-loss and the fifth sub-loss.

[0063] Among them, the first brightness image can be understood as a synthesized image, and the brightness value of each pixel in the first brightness image is the larger value of the brightness value of the pixel in the sample visible light image and the sample infrared image, in order to ensure that higher brightness information can be retained. The third sub-loss can be understood as a loss value calculated based on the difference between the first sample fused image and the first brightness image, reflecting the ability of the first fused image in brightness retention. The second brightness image can be understood as a synthesized image, and the brightness value of each pixel in the second brightness image is the larger value of the brightness value of the pixel in the gradient image of the sample visible light image and the gradient image of the sample infrared image, in order to ensure that the fused image can retain higher gradient information, including edges and details. The fourth sub-loss can be understood as a loss value calculated based on the difference between the gradient image of the first sample fused image and the second brightness image, reflecting the performance of the fused image in maintaining edges and details. The third sample fused image can be a fused image obtained by multiplying the sample visible light image and the sample infrared image, in order to emphasize the features that exist in both the sample visible light image and the sample infrared image. The fifth sub-loss may be a loss value calculated according to the difference between the second sample fused image and the third sample fused image, reflecting the performance in maintaining the common features of the two original images.

[0064] Based on the above solution, optionally, a third sub-loss between the first sample fused image and the first brightness image is determined based on the following formula:

[0065]

[0066] Among them, L int represents the third sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, I ir Represents the sample visible light image, I vis Represents the sample infrared image, max(I ir , I vis ) represents the first brightness image, ∥·∥1 is a function for calculating Manhattan distance;

[0067] As an optional implementation of the embodiment of the present invention, the fourth sub-loss between the gradient image of the first sample fused image and the second brightness image may be determined based on the following formula:

[0068]

[0069] Among them, Ltex represents the fourth sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, represents the gradient image of the first sample fusion image, I ir represents the sample visible light image, Represents the gradient image of the sample visible light image, I vis represents a sample infrared image, represents the gradient image of the sample infrared image, represents the second brightness image, and ∥.∥1 is the function for calculating the Manhattan distance.

[0070] The image height and the image width represent the height and width of the image, respectively, and can be understood as the size of the sample visible light image or the sample infrared image, and are used as normalization factors when calculating the loss value.

[0071] As an optional implementation of an embodiment of the present invention, a first brightness image can be calculated based on the sample visible light image and the sample infrared image, and then the Manhattan distance between the first sample fusion image and the first brightness image is calculated and normalized to obtain a third sub-loss.

[0072] As an optional implementation of an embodiment of the present invention, a gradient image can be calculated based on the sample visible light image and the sample infrared image to obtain a second brightness image, thereby calculating the Manhattan distance between the gradient image of the first sample fusion image and the second brightness image, and normalizing them to obtain a fourth sub-loss.

[0073] S240. Acquire a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image.

[0074] S250: Determine a third model fusion image according to the first model fusion image and the second model fusion image, and superimpose the first model fusion image and the third model fusion image to obtain a target fusion image.

[0075] The technical solution of the embodiment of the present invention optimizes the fusion effect of visible light and infrared images through multi-scale fusion and feature extraction. First, the model is trained using sample images to generate different fused images, and the feature extraction process is optimized using the target feature extraction network; second, the brightness and gradient images are introduced to assist in calculating the loss function, so that the model can be optimized for the visual characteristics of the image during training, thereby generating a more visually natural and information-rich fused image.

[0076] Figure 3 A flowchart of a method for fusing a visible light image and an infrared image provided in Embodiment 3 of the present invention. The technical solution of this embodiment is based on the above embodiment, and further refines how to determine the second loss according to the target features corresponding to the third sample fused image, the sample visible light image and the sample infrared image. Optionally, the target feature extraction network includes a texture feature extraction network and a semantic feature extraction network; the first target feature corresponding to the texture feature extraction network includes a first texture feature, the second target feature includes a second texture feature, and the third target feature includes a third texture feature; the first target feature corresponding to the semantic feature extraction network includes a first semantic feature, the second target feature includes a second semantic feature, and the third target feature includes a third semantic feature; determining the first loss according to the target features corresponding to the third sample fusion image, the sample visible light image, and the sample infrared image includes: determining a first sub-loss according to determining a first texture sub-loss between the first texture feature and the second texture feature and a second texture sub-loss between the first texture feature and the third texture feature; determining a second sub-loss according to determining a first semantic sub-loss between the first semantic feature and the second semantic feature and a second semantic sub-loss between the first semantic feature and the third semantic feature; determining a first loss according to the first sub-loss and the second sub-loss.

[0077] The specific implementation method can refer to the description of this embodiment, wherein the technical features that are the same or similar to the above-mentioned embodiments are not described in detail here.

[0078] like Figure 3 As shown, the method may specifically include:

[0079] S310, obtaining a sample visible light image and a sample infrared image, inputting the sample visible light image and the sample infrared image into a model to be trained to obtain a first sample fusion image and a second sample fusion image, and determining a third sample fusion image according to the first sample fusion image and the second sample fusion image.

[0080] S320. Extract, through at least one target feature extraction network, a first target feature corresponding to the third sample fusion image, a second target feature corresponding to the sample visible light image, and a third target feature corresponding to the sample infrared image.

[0081] The target feature extraction network includes a texture feature extraction network and a semantic feature extraction network; the first target feature corresponding to the texture feature extraction network includes a first texture feature, the second target feature includes a second texture feature, and the third target feature includes a third texture feature; the first target feature corresponding to the semantic feature extraction network includes a first semantic feature, the second target feature includes a second semantic feature, and the third target feature includes a third semantic feature.

[0082] Among them, the first texture feature may be a texture feature extracted from the third sample fused image. The second texture feature may be a texture feature extracted from the enhanced image of the sample visible light image. The third texture feature may be a texture feature extracted from the sample infrared image. The third texture feature is different from the second texture feature in the visible light image. Extracting the third texture feature helps to increase the infrared characteristics during fusion. Correspondingly, the first semantic feature may be a semantic feature extracted from the third sample fused image. The second semantic feature may be a semantic feature extracted from the enhanced image of the sample visible light image. The third semantic feature may be a semantic feature extracted from the sample infrared image.

[0083] Through the above scheme, not only can the performance of the fused image in texture preservation be evaluated, but it can also ensure that the fused image retains the semantic features of the original image to guide the adjustment of model parameters and optimize the effect of image fusion.

[0084] S330: Determine a first sub-loss according to a first texture sub-loss between the first texture feature and the second texture feature and a second texture sub-loss between the first texture feature and the third texture feature.

[0085] Based on the above solution, optionally, the first sub-loss is determined according to the first texture sub-loss between the first texture feature and the second texture feature and the second texture sub-loss between the first texture feature and the third texture feature based on the following formula:

[0086]

[0087] Among them, L Gabor represents the first sub-loss, I xor represents the third sample fused image, f Gabor (I xor) represents the first texture feature obtained after the third sample fusion image is input into the texture feature extraction model, represents the enhanced image of the sample visible light image, It represents the second texture feature obtained by inputting the enhanced image of the sample visible light image into the texture feature extraction model, I ir represents the sample infrared image, f Gabor (I ir ) represents the third texture feature obtained after inputting the sample infrared image into the texture feature extraction model, and ∥.∥1 is the function for calculating the Manhattan distance.

[0088] The first texture sub-loss can be understood as a loss value determined according to the difference between the first texture feature and the second texture feature, reflecting the performance of the third sample fused image and the sample visible light image in terms of texture preservation. The second texture sub-loss can be understood as a loss value determined according to the difference between the first texture feature and the third texture feature, reflecting the performance of the third sample fused image and the sample infrared image in terms of texture preservation. The texture feature extraction model can be used to identify and extract texture features in an image, so as to quantify the difference in texture preservation between the third sample fused image and the sample visible light image and the sample infrared image by calculating the first texture sub-loss and the second texture sub-loss.

[0089] S340: Determine a second sub-loss based on a first semantic sub-loss between the first semantic feature and the second semantic feature and a second semantic sub-loss between the first semantic feature and the third semantic feature.

[0090] Based on the above solution, optionally, the second sub-loss is determined based on the following formula according to the first semantic sub-loss between the first semantic feature and the second semantic feature and the second semantic sub-loss between the first semantic feature and the third semantic feature:

[0091]

[0092] Among them, L VGG represents the second sub-loss, I xor represents the third sample fused image, f VGG (I xor ) represents the first semantic feature obtained after the third sample fusion image is input into the semantic feature extraction model, represents the enhanced image of the sample visible light image, It represents the second semantic feature obtained by inputting the enhanced image of the sample visible light image into the semantic feature extraction model, I ir represents the sample infrared image, f VGG (I ir) represents the third semantic feature obtained after inputting the sample infrared image into the semantic feature extraction model, and ∥.∥1 is the function for calculating the Manhattan distance

[0093] Among them, the first semantic sub-loss can be understood as a loss value determined according to the difference between the first semantic feature and the second semantic feature, reflecting the performance of the third sample fused image and the sample visible light image in terms of semantic preservation. The second semantic sub-loss can be understood as a loss value determined according to the difference between the first semantic feature and the third semantic feature, reflecting the performance of the third sample fused image and the sample infrared image in terms of semantic preservation. The semantic feature extraction model can be used to identify and extract semantic features in an image to quantify the difference in semantic preservation between the third sample fused image and the sample visible light image and the sample infrared image by calculating the first semantic sub-loss and the second semantic sub-loss.

[0094] S350: Determine a first loss according to the first sub-loss and the second sub-loss.

[0095] S360. Determine a target loss according to the first loss, and adjust parameters of the model to be trained according to the target loss to obtain an image fusion model.

[0096] S370. Acquire a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image.

[0097] S380: Determine a third model fusion image according to the first model fusion image and the second model fusion image, and superimpose the first model fusion image and the third model fusion image to obtain a target fusion image.

[0098] The technical solution of the embodiment of the present invention optimizes the image fusion model by combining the texture feature extraction network and the semantic feature extraction network, and utilizes the loss calculation of texture and semantic features to ensure that the fused image can achieve high quality in both details and semantics; in addition, by enhancing the input of the sample visible light image, the model's ability to capture the features of the visible light image is improved, making the fusion result richer and more accurate, improving the visual effect of the image fusion, and enhancing the versatility and practicality of the model.

[0099] Embodiment 3

[0100] Figure 4 The method for fusing a visible light image and an infrared image provided in the third embodiment of the present invention is an optional embodiment of the above embodiment.

[0101] like Figure 4 As shown, the method may specifically include:

[0102] Firstly, a sample visible light image and a sample infrared image are input into the model to be trained, and the model to be trained is divided into two training branches, one training branch outputs a first sample fusion image, and the other training branch outputs a second sample fusion image;

[0103] Next, the second loss of the sample visible light image, the sample infrared image, the first sample fusion image, and the second sample fusion image is determined by the following steps:

[0104] 1) Determine the maximum intensity of the sample infrared image and the sample visible light image as the first brightness image, and determine the third sub-loss of the first brightness image and the first sample fusion image:

[0105]

[0106] Among them, L int represents the first sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, I ir Represents the sample visible light image, I vis Represents the sample infrared image, max(I ir , I vis ) represents the first brightness image, ∥.∥1 is a function for calculating Manhattan distance;

[0107] 2) Determine the maximum value of the gradient image of the sample infrared image and the gradient image of the sample visible light image as the second brightness image, and determine the fourth sub-loss of the gradient image of the second brightness image and the first sample fusion image:

[0108]

[0109] Among them, L tex represents the second sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, represents the gradient image of the first sample fusion image, I ir represents the sample visible light image, Represents the gradient image of the sample visible light image, I vis represents a sample infrared image, represents the gradient image of the sample infrared image, represents the second brightness image, ∥·∥1 is the function for calculating the Manhattan distance;

[0110] 3) Multiply the sample visible light image and the sample infrared image, perform normalization, determine the third sample fusion image, and determine the fifth sub-loss between the second sample fusion image and the third sample fusion image:

[0111] I xor = rescale(I fin +I feg -2I fin I feg ),

[0112] Among them, I xor represents the third model fusion image, I fin represents the first model fusion image, I feg Represents the second model fusion image, and rescale(.) is a function that scales the input image to [0,1].

[0113] 4) The third sub-loss, the fourth sub-loss and the fifth sub-loss are weightedly summed to determine the second loss.

[0114] Then, the first loss of the target features corresponding to the third sample fusion image, the sample visible light image and the sample infrared image is determined by the following steps:

[0115] 1) Using a Gabor feature extractor to extract texture features of the third sample fusion image, the enhanced image of the sample visible light image, and the sample infrared image, and determining a first sub-loss, wherein the enhanced image of the sample visible light image can be normalized by a grayscale histogram:

[0116]

[0117] Among them, L Gabor represents the first sub-loss, I xor represents the third sample fused image, f Gabor (I xor ) represents the first texture feature obtained after the third sample fusion image is input into the texture feature extraction model, represents the enhanced image of the sample visible light image, It represents the second texture feature obtained by inputting the enhanced image of the sample visible light image into the texture feature extraction model, I ir represents the sample infrared image, f Cabor (I ir) represents the third texture feature obtained after inputting the sample infrared image into the texture feature extraction model, and ∥.∥1 is the function for calculating the Manhattan distance.

[0118] 2) Using the VGG feature extractor, the semantic features of the third sample fusion image, the enhanced image of the sample visible light image, and the sample infrared image are extracted to determine the second sub-loss:

[0119]

[0120] Among them, L VGG represents the second sub-loss, I xor represents the third sample fused image, f VGG (I xor ) represents the first semantic feature obtained after the third sample fusion image is input into the semantic feature extraction model, represents the enhanced image of the sample visible light image, It represents the second semantic feature obtained by inputting the enhanced image of the sample visible light image into the semantic feature extraction model, I ir represents the sample infrared image, f VGG (I ir ) represents the third semantic feature obtained after the sample infrared image is input into the semantic feature extraction model, and ∥·∥1 is the function for calculating the Manhattan distance.

[0121] 3) The first loss is determined by weighted summing the first sub-loss and the second sub-loss.

[0122] Next, the first loss and the second loss are weightedly summed to determine the target loss of the model to be trained, and the parameters of the model to be trained are adjusted to obtain an image fusion model.

[0123] Then, the target visible light image and the target infrared image are input into the image fusion model to obtain a first model fusion image and a second model fusion image;

[0124] A third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image.

[0125] The technical solution of this embodiment firstly generates multiple fused images by different processing of sample images (such as brightness and gradient analysis) and multi-branch models to ensure the quality of fused images; then, Gabor feature extractor and VGG feature extractor are used to capture texture and semantic information respectively to comprehensively evaluate the fusion effect. By integrating multiple loss functions such as texture, semantics and brightness gradient, the information richness of image fusion can be effectively improved, and the final output fused image has clear details and can better adapt to complex environmental conditions.

[0126] Embodiment 4

[0127] Figure 5 This is a schematic diagram of the structure of a visible light image and infrared image fusion device provided in Embodiment 4 of the present invention. Figure 5 As shown, the device may include: a first image fusion module 510 and a second image fusion module 520.

[0128] Among them, the first image fusion module 510 is used to obtain a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image; the second image fusion module 520 is used to determine a third model fusion image based on the first model fusion image and the second model fusion image, and superimpose the first model fusion image and the third model fusion image to obtain a target fusion image.

[0129] The technical solution of the embodiment of the present invention is as follows: first, a target visible light image and a target infrared image are obtained through a first image fusion module 510, and the target visible light image and the target infrared image are input into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image, wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image, and the first model fusion image and the second model fusion image are respectively generated by two different fusion networks, which will highlight the details and features of the visible light image and the infrared image, thereby providing a more diverse image information display method and effectively enhancing the image quality and information content; then, a third model fusion image is determined according to the first model fusion image and the second model fusion image through a second image fusion module 520, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image. By generating and superimposing multiple fusion images, the feature types and information amount of the target fusion image can be increased, and the visual effect of the overall image can be improved. The technical solution of the embodiment of the present invention generates a first model fusion image and a second model fusion image respectively through two independent fusion networks, then determines a third model fusion image, and then superimposes the first and third model fusion images to obtain a target fusion image, which solves the problems of information loss of visible light images and infrared images, high feature accuracy requirements and complex parameter adjustment in related technologies, enhances the key information in the fused image, and is conducive to accurately identifying and positioning targets under harsh conditions such as high light and smoke.

[0130] On the basis of the above solution, optionally, the visible light image and infrared image fusion device further includes: a sample fusion image determination module, a first loss determination module, and an image fusion model determination module.

[0131] Among them, the sample fusion image determination module is used to obtain a sample visible light image and a sample infrared image before inputting the target visible light image and the target infrared image into the pre-trained image fusion model, input the sample visible light image and the sample infrared image into the model to be trained to obtain a first sample fusion image and a second sample fusion image, and determine a third sample fusion image according to the first sample fusion image and the second sample fusion image; the first loss determination module is used to extract the first target feature corresponding to the third sample fusion image, the second target feature corresponding to the sample visible light image, and the third target feature corresponding to the sample infrared image through at least one target feature extraction network, and determine a first loss according to the first target feature, the second target feature, and the third target feature, wherein the target feature extraction network includes a texture feature extraction network and / or a semantic feature extraction network; the image fusion model determination module is used to determine a target loss according to the first loss, and adjust the parameters of the model to be trained according to the target loss to obtain an image fusion model.

[0132] On the basis of the above scheme, optionally, the target feature extraction network includes a texture feature extraction network and a semantic feature extraction network; the first target feature corresponding to the texture feature extraction network includes a first texture feature, the second target feature includes a second texture feature, and the third target feature includes a third texture feature; the first target feature corresponding to the semantic feature extraction network includes a first semantic feature, the second target feature includes a second semantic feature, and the third target feature includes a third semantic feature; the first loss determination module may include: a first sub-loss determination sub-module, a second sub-loss determination sub-module, and a first loss determination sub-module.

[0133] Among them, the first sub-loss determination submodule is used to determine the first sub-loss according to the first texture sub-loss between the first texture feature and the second texture feature and the second texture sub-loss between the first texture feature and the third texture feature; the second sub-loss determination submodule is used to determine the second sub-loss according to the first semantic sub-loss between the first semantic feature and the second semantic feature and the second semantic sub-loss between the first semantic feature and the third semantic feature; the first loss determination submodule is used to determine the first loss according to the first sub-loss and the second sub-loss.

[0134] On the basis of the above solution, further, the first loss determination submodule is specifically used to determine the first sub-loss according to the first texture sub-loss between the first texture feature and the second texture feature and the second texture sub-loss between the first texture feature and the third texture feature based on the following formula:

[0135]

[0136] Among them, L Gabor represents the first sub-loss, I xor represents the third sample fused image, f Gabor (I xor ) represents the first texture feature obtained after the third sample fusion image is input into the texture feature extraction model, represents the enhanced image of the sample visible light image, It represents the second texture feature obtained by inputting the enhanced image of the sample visible light image into the texture feature extraction model, I ir represents the sample infrared image, f Gabor (I ir ) represents the third texture feature obtained after inputting the sample infrared image into the texture feature extraction model, and ∥·∥1 is a function for calculating the Manhattan distance.

[0137] On the basis of the above scheme, further, the second loss determination submodule is specifically used to determine the second sub-loss according to the first semantic sub-loss between the first semantic feature and the second semantic feature and the second semantic sub-loss between the first semantic feature and the third semantic feature based on the following formula:

[0138]

[0139] Among them, L VGG represents the second sub-loss, I xor represents the third sample fused image, f VGG (I xor ) represents the first semantic feature obtained after the third sample fusion image is input into the semantic feature extraction model, represents the enhanced image of the sample visible light image, It represents the second semantic feature obtained by inputting the enhanced image of the sample visible light image into the semantic feature extraction model, I ir represents the sample infrared image, f VGG (I ir ) represents the third semantic feature obtained after inputting the sample infrared image into the semantic feature extraction model, and ∥.∥1 is the function for calculating the Manhattan distance.

[0140] Based on the above solution, optionally, the visible light image and infrared image fusion device further includes: a second loss determination module. The second loss determination module is used to determine the second loss of the model to be trained according to the sample visible light image, the sample infrared image, the first sample fusion image and the second sample fusion image after obtaining the first sample fusion image and the second sample fusion image.

[0141] Based on the above solution, optionally, the image fusion model determination module may include: a target loss submodule. The target loss submodule is used to determine the target loss according to the first loss and the second loss.

[0142] On the basis of the above scheme, optionally, the second loss determination module is specifically used to determine a first brightness image according to the sample visible light image and the sample infrared image, and determine a third sub-loss between the first sample fusion image and the first brightness image; wherein the brightness value of each pixel in the first brightness image is the larger value of the brightness values ​​of the pixel in the sample visible light image and the sample infrared image; determine a second brightness image according to the gradient image of the sample visible light image and the gradient image of the sample infrared image, and determine a fourth sub-loss between the gradient image of the first sample fusion image and the second brightness image; wherein the brightness value of each pixel in the second brightness image is the larger value of the brightness values ​​of the pixel in the gradient image of the sample visible light image and the gradient image of the sample infrared image; multiply the sample visible light image and the sample infrared image to obtain a third sample fusion image, determine a fifth sub-loss between the second sample fusion image and the third sample fusion image, and determine the second loss of the model to be trained according to the third sub-loss, the fourth sub-loss and the fifth sub-loss.

[0143] On the basis of the above solution, further, the second loss determination module may include: a third sub-loss determination sub-module and a fourth sub-loss determination sub-module.

[0144] The third sub-loss determination submodule is used to determine the third sub-loss between the first sample fusion image and the first brightness image based on the following formula:

[0145]

[0146] Among them, L int represents the third sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I finrepresents the first sample fused image, I ir Represents the sample visible light image, I vis Represents the sample infrared image, max(I ir， I vis ) represents the first brightness image, ∥·∥1 is a function for calculating Manhattan distance;

[0147] and / or,

[0148] The fourth sub-loss determination submodule is used to determine the fourth sub-loss between the gradient image of the first sample fusion image and the second brightness image based on the following formula:

[0149]

[0150] Among them, L tex represents the fourth sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, represents the gradient image of the first sample fusion image, I ir represents the sample visible light image, Represents the gradient image of the sample visible light image, I vis represents a sample infrared image, represents the gradient image of the sample infrared image, represents the second brightness image, and ∥.∥1 is the function for calculating the Manhattan distance.

[0151] On the basis of the above scheme, optionally, the first loss determination module may include: a first target feature determination submodule, a second target feature determination submodule, and a third target feature determination submodule.

[0152] Among them, the first target feature determination submodule is used to input the third sample fusion image into the target feature extraction model to obtain a first target feature corresponding to the third sample fusion image; the second target feature determination submodule is used to input the enhanced image of the sample visible light image into the target feature extraction model to obtain a second target feature corresponding to the sample visible light image; and the third target feature determination submodule is used to input the sample infrared image into the target feature extraction model to obtain a third target feature corresponding to the sample infrared image.

[0153] Based on the above solution, optionally, the second image fusion module may include: a third model fusion image determination submodule. The third model fusion image determination submodule is used to determine the third model fusion image according to the first model fusion image and the second model fusion image based on the following formula, including:

[0154] I xor = rescale(I fin +I feg -2I fin I feg ),

[0155] Among them, I xor represents the third model fusion image, I fin represents the first model fusion image, I feg represents the fused image of the second model, and rescale(·) is a function that scales the input image to [0,1].

[0156] The visible light image and infrared image fusion device provided in the embodiment of the present invention can execute the visible light image and infrared image fusion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0157] Embodiment 5

[0158] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0159] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for fusing visible light images and infrared images.

[0162] In some embodiments, the method for fusing visible light images and infrared images may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for fusing visible light images and infrared images described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method for fusing visible light images and infrared images in any other appropriate manner (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fusing a visible light image and an infrared image, characterized in that: include: Acquire a target visible light image and a target infrared image, and input the target visible light image and the target infrared image into a pre-trained image fusion model to obtain a first model fusion image and a second model fusion image; wherein the image fusion model includes a first fusion network corresponding to the first model fusion image and a second fusion network corresponding to the second model fusion image; A third model fusion image is determined according to the first model fusion image and the second model fusion image, and the first model fusion image and the third model fusion image are superimposed to obtain a target fusion image.

2. The method according to claim 1, characterized in that Before inputting the target visible light image and the target infrared image into the pre-trained image fusion model, the method further includes: Acquire a sample visible light image and a sample infrared image, input the sample visible light image and the sample infrared image into a model to be trained to obtain a first sample fusion image and a second sample fusion image, and determine a third sample fusion image according to the first sample fusion image and the second sample fusion image; respectively extracting a first target feature corresponding to the third sample fusion image, a second target feature corresponding to the sample visible light image, and a third target feature corresponding to the sample infrared image through at least one target feature extraction network, and determining a first loss according to the first target feature, the second target feature, and the third target feature, wherein the target feature extraction network includes a texture feature extraction network and / or a semantic feature extraction network; A target loss is determined according to the first loss, and parameters of the model to be trained are adjusted according to the target loss to obtain an image fusion model.

3. The method according to claim 2, characterized in that The target feature extraction network includes a texture feature extraction network and a semantic feature extraction network; the first target feature corresponding to the texture feature extraction network includes a first texture feature, the second target feature includes a second texture feature, and the third target feature includes a third texture feature; the first target feature corresponding to the semantic feature extraction network includes a first semantic feature, the second target feature includes a second semantic feature, and the third target feature includes a third semantic feature; Determining a first loss according to target features corresponding to the third sample fusion image, the sample visible light image, and the sample infrared image includes: Determine a first sub-loss according to a first texture sub-loss between the first texture feature and the second texture feature and a second texture sub-loss between the first texture feature and the third texture feature; determining a second sub-loss according to a first semantic sub-loss between the first semantic feature and the second semantic feature and a second semantic sub-loss between the first semantic feature and the third semantic feature; A first loss is determined according to the first sub-loss and the second sub-loss.

4. The method according to claim 3, characterized in that A first sub-loss is determined based on the following formula according to determining a first texture sub-loss between the first texture feature and the second texture feature and a second texture sub-loss between the first texture feature and the third texture feature: L Gabor =||f Gabor (I xor )-f Gabor (Iviseql)||1+||f Gabor (I XOR )-f Gabor (I ir )||1, Among them, L Gabor represents the first sub-loss, I xor represents the third sample fused image, f Gabor (I xor ) represents the first texture feature obtained after the third sample fusion image is input into the texture feature extraction model, represents the enhanced image of the sample visible light image, It represents the second texture feature obtained by inputting the enhanced image of the sample visible light image into the texture feature extraction model, I ir represents the sample infrared image, f Gabor (I ir ) represents the third texture feature obtained after inputting the sample infrared image into the texture feature extraction model, and ||.||1 is the function for calculating the Manhattan distance.

5. The method according to claim 3, characterized in that: The second sub-loss is determined based on the following formula according to the first semantic sub-loss between the first semantic feature and the second semantic feature and the second semantic sub-loss between the first semantic feature and the third semantic feature: L VGG =||f VGG (I xor )-f VGG (I viseql )||1+||f VGG (I xor )-f VGG (I ir )||1, Among them, L VGG represents the second sub-loss, I xor represents the third sample fused image, f VGG (I xor ) represents the first semantic feature obtained after the third sample fusion image is input into the semantic feature extraction model, represents the enhanced image of the sample visible light image, It represents the second semantic feature obtained by inputting the enhanced image of the sample visible light image into the semantic feature extraction model, I ir represents the sample infrared image, f VGG (I ir ) represents the third semantic feature obtained after inputting the sample infrared image into the semantic feature extraction model, and ∥.∥1 is the function for calculating the Manhattan distance.

6. The method according to claim 2, characterized in that After obtaining the first sample fusion image and the second sample fusion image, the method further includes: Determine a second loss of the to-be-trained model according to the sample visible light image, the sample infrared image, the first sample fused image, and the second sample fused image; The determining the target loss according to the first loss includes: A target loss is determined according to the first loss and the second loss.

7. The method according to claim 6, characterized in that The determining the second loss of the to-be-trained model according to the sample visible light image, the sample infrared image, the first sample fusion image, and the second sample fusion image comprises: Determine a first brightness image according to the sample visible light image and the sample infrared image, and determine a third sub-loss between the first sample fused image and the first brightness image; wherein the brightness value of each pixel in the first brightness image is the larger value of the brightness values ​​of the pixel in the sample visible light image and the sample infrared image; Determine a second brightness image according to the gradient image of the sample visible light image and the gradient image of the sample infrared image, and determine a fourth sub-loss between the gradient image of the first sample fused image and the second brightness image; wherein the brightness value of each pixel in the second brightness image is a larger value of the brightness values ​​of the pixel in the gradient image of the sample visible light image and the gradient image of the sample infrared image; The sample visible light image and the sample infrared image are multiplied to obtain a third sample fusion image, a fifth sub-loss between the second sample fusion image and the third sample fusion image is determined, and a second loss of the model to be trained is determined according to the third sub-loss, the fourth sub-loss and the fifth sub-loss.

8. The method according to claim 7, characterized in that: The third sub-loss between the first sample fused image and the first brightness image is determined based on the following formula: Among them, L int represents the third sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, I ir Represents the sample visible light image, I vis Represents the sample infrared image, max(I ir， I vis ) represents the first brightness image, ||.||1 is a function for calculating Manhattan distance; and / or, The fourth sub-loss between the gradient image of the first sample fused image and the second brightness image is determined based on the following formula: Among them, L tex represents the fourth sub-loss, H represents the image height, W represents the image width, the image height is the height of the sample visible light image or the sample infrared image, the image width is the width of the sample visible light image or the sample infrared image, I fin represents the first sample fused image, represents the gradient image of the first sample fusion image, I ir represents the sample visible light image, Represents the gradient image of the sample visible light image, I vis represents a sample infrared image, represents the gradient image of the sample infrared image, represents the second brightness image, and ||·||1 is the function for calculating the Manhattan distance.

9. The method according to claim 2, characterized in that: The extracting, by at least one target feature extraction network, the first target feature corresponding to the third sample fusion image, the second target feature corresponding to the sample visible light image, and the third target feature corresponding to the sample infrared image respectively includes: The third sample fusion image is input into the target feature extraction model to obtain a first target feature corresponding to the third sample fusion image; the enhanced image of the sample visible light image is input into the target feature extraction model to obtain a second target feature corresponding to the sample visible light image; and the sample infrared image is input into the target feature extraction model to obtain a third target feature corresponding to the sample infrared image.

10. The method according to claim 1, characterized in that Determining a third model fusion image according to the first model fusion image and the second model fusion image based on the following formula includes: I xor =rescale(I fin +I feg -2I fin ·I feg ), Among them, I xor represents the third model fusion image, I fin represents the first model fusion image, I feg Represents the second model fusion image, and rescale(.) is a function that scales the input image to [0,1].

Images