Multi-sensor adaptive fusion imaging method, device, equipment and medium

Abstract

The invention discloses a multi-sensor adaptive fusion imaging method, device, equipment and medium, and belongs to the sensor fusion imaging technology, and the method comprises the steps: obtaining an infrared image and a visible light image corresponding to the infrared image, constructing an image pair, and constructing a data set according to the image pair; performing data enhancement processing on the constructed data set to obtain an enhanced data set; and constructing an image generation model, and training the image generation model by using the enhanced data set. According to the method, a multi-sensor feature selection module is constructed in the model, and important features are adaptively selected for fusion and noise corresponding to each sensor is suppressed by learning mask information corresponding to different sensor information, so that the robustness of an algorithm to the noise is improved, and meanwhile, the computing resource demand of the model is reduced. The system can be applied to vehicles such as vehicles and airplanes to assist drivers in acquiring surrounding environment information, so that driving safety in rainy and foggy days is guaranteed.

Description

Technical Field

[0001] The present invention relates to sensor fusion imaging technology, and in particular to a multi-sensor adaptive fusion imaging method, device, equipment and medium. Background Art

[0002] Sensor fusion methods have achieved significant breakthroughs in computer vision fields such as image classification, target detection, video analysis, and semantic segmentation, especially in the field of fusion imaging assisted driving. Due to the complementary characteristics of infrared and visible light images, infrared imaging based on sensor fusion has become a research hotspot. However, the sensor data used in the sensor fusion method has large modal differences and is easily affected by the environment, resulting in low fusion imaging quality, making it difficult to directly apply the sensor fusion method to the real production field. For example, the data used by the sensor will be affected by rain, fog, noise, etc. in the real environment, resulting in low imaging quality and cannot be directly applied.

[0003] In order to improve the quality of infrared imaging, the research on sensor fusion mainly focuses on how to utilize the complementary characteristics of infrared and visible light images to obtain robust and information-rich infrared imaging. Existing methods are generally based on learnable deep neural networks, which are highly adaptable and resistant to noise. However, the existing deep neural network fusion methods ignore the large differences in sensor data modalities and the susceptibility to environmental influences during modeling, resulting in the inability to directly apply imaging to real rainy and foggy environments. In addition, the use of single-sensor imaging in some environments can also meet established requirements and avoid unnecessary computational burdens. For example, in a bright, unobstructed environment, the quality of single-sensor imaging is high enough, and there is no need to fuse infrared sensors. However, existing methods are not sensitive to the environment, making it difficult to apply sensor fusion algorithms to devices with limited computing resources, such as cars, aircraft and other driving equipment. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a multi-sensor adaptive fusion imaging method, device, equipment and medium based on infrared and visible light cameras.

[0005] The first technical solution adopted by the present invention is:

[0006] A multi-sensor adaptive fusion imaging method comprises the following steps:

[0007] Acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair;

[0008] Perform data enhancement processing on the constructed data set to obtain an enhanced data set;

[0009] An image generation model is constructed and trained using the enhanced data set; the image generation model takes the infrared image and the obstructed visible light image as input and outputs an unobstructed visible light image;

[0010] Among them, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning the mask information corresponding to different sensors, and suppress the noise corresponding to each sensor, thereby improving the robustness of the algorithm to noise.

[0011] Furthermore, the acquiring of the infrared image and the visible light image corresponding to the infrared image to construct an image pair includes:

[0012] For the same environment, an infrared image is acquired through an infrared sensor, and a visible light image is acquired through a visible light sensor, and an image pair P = {A, B} is constructed to represent different features of the same environment;

[0013] The infrared image A consists of M×N points {p1, p2, …, p M×N}, each point p i Represents temperature information; the visible light image B consists of M×N points {p1, p2, …, p M×N}, each point p i Represents the color information of the three channels of red, green and blue.

[0014] Furthermore, performing data enhancement processing on the constructed data set to obtain an enhanced data set includes:

[0015] Use a pre-trained deep neural network to calculate the temperature detail mask C from the infrared image A and the texture detail mask D from the visible light image B to enhance the dataset;

[0016] Adding preset noise to the visible light image using a preset noise image to obtain an obscured visible light image E; wherein the preset noise includes rain and fog noise;

[0017] Among them, the enhanced image pair is represented as P A ={A,B,C,D,E}.

[0018] Furthermore, the generator works as follows:

[0019] Input the enhanced image pair P to the multi-sensor feature selection module S A ={A, B, C, D, E} infrared image A and the obscured visible light image E, generate infrared image mask M Aand visible light image mask M E ;

[0020] According to the generated mask, the infrared image A and the obscured visible light image E are masked to generate image A. m and image E m :

[0021] A m =A⊙M A ,

[0022] E m =E⊙M E ,

[0023] Where ⊙ represents the matrix Hadamard product;

[0024] Input the masked image A to the generator Gen m and image E m , generate the fused image G.

[0025] Furthermore, the discriminator works as follows:

[0026] According to the enhanced image P A = image mask C, D of {A, B, C, D, E}, perform mask operation on infrared image A, visible light image B and fused image G to generate masked image A C ,B D ,G C ,G D :

[0027] A C =A⊙C,

[0028] B D =B⊙D,

[0029] G C =G⊙C,

[0030] G D =G⊙D,

[0031] To the discriminator Dis T Input image A C and image G C , get the discrimination probability T real ,T fake ;

[0032] To the discriminator Dis D Input image B D and image G D , get the discrimination probability D real ,D fale ;

[0033] Calculate the adversarial loss based on the obtained discriminant probability

[0034]

[0035] Furthermore, the step of training the image generation model includes the step of optimizing the generator and the discriminator:

[0036] Based on the obtained adversarial loss Then, according to the structural similarity loss SSIM(A,G) and SSIM(B,G) of the fused image G and the infrared image A, and the fused image G and the visible light image B, the generator loss value is calculated.

[0037]

[0038] In the formula, SSIM(,) represents the structural similarity loss;

[0039] According to the adversarial loss Calculate the discriminator loss value separately

[0040]

[0041] The generator, multi-sensor feature selection module, and discriminator are optimized according to the obtained loss values.

[0042] Furthermore, the multi-sensor feature selection module works as follows:

[0043] Acquire an infrared image and a visible light image acquired by a sensor, wherein the infrared image and the visible light image form an image pair;

[0044] Input the infrared image and visible light image to the multi-sensor feature selection module S, and calculate the infrared image mask M respectively. A and visible light image mask M B The average intensity produces the selectivity coefficient S of the infrared image A and the selectivity coefficient S of the visible light image B :

[0045] S A =avg(M A ),

[0046] S B =avg(M B ),

[0047] Where avg(M) represents the average value of the information flow intensity calculated for all data points of mask M;

[0048] If and only if the selectivity coefficient S of the infrared imageA Greater than the selectivity coefficient S of the visible light image B When , the infrared and visible light images are input to the generator Gen to generate the fused image G, otherwise the visible light image B is directly output.

[0049] The second technical solution adopted by the present invention is:

[0050] A multi-sensor adaptive fusion imaging device, comprising:

[0051] A data acquisition module, used to acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair;

[0052] A data enhancement module is used to perform data enhancement processing on the constructed data set to obtain an enhanced data set;

[0053] A model training module is used to construct an image generation model and train the image generation model using the enhanced data set; the image generation model takes the infrared image and the obscured visible light image as input and outputs an unobstructed visible light image;

[0054] Among them, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning mask information corresponding to different sensor information, and suppress the noise corresponding to each sensor, thereby improving the algorithm's robustness to noise.

[0055] The third technical solution adopted by the present invention is:

[0056] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor adaptive fusion imaging method as described above.

[0057] The fourth technical solution adopted by the present invention is:

[0058] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.

[0059] The fifth technical solution adopted by the present invention is:

[0060] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0061] The beneficial effect of the present invention is that the present invention constructs a multi-sensor feature selection module in the model, adaptively selects important features for fusion and suppresses the noise corresponding to each sensor by learning the mask information corresponding to different sensor information, thereby improving the robustness of the algorithm to noise and reducing the computing resource requirements of the model. It can be applied to vehicles, airplanes and other means of transportation to assist drivers in obtaining information about the surrounding environment, thereby ensuring driving safety in rainy and foggy days. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 This is a flowchart of the steps of a multi-sensor adaptive fusion imaging method based on infrared and visible light cameras in an embodiment of the present invention;

[0064] Figure 2 is a schematic diagram of an enhanced data pair in an embodiment of the present invention;

[0065] Figure 3 is a schematic diagram of generator imaging in an embodiment of the present invention;

[0066] Figure 4 is a schematic diagram of the discriminator discrimination in an embodiment of the present invention;

[0067] Figure 5 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0069] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0070] In the description of the present invention, the meaning of "several" is one or more, the meaning of "more" is two or more, and the meanings of "greater than", "less than", "exceed" and the like are understood as not including the number itself, and the meanings of "above", "below", "within" and the like are understood as including the number itself. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0071] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0072] In view of the problem that the existing methods ignore the large modal differences of sensor data, are easily affected by rain and fog noise, and are not sensitive to environmental rain and fog noise when modeling. The present invention provides an adaptive sensor fusion imaging solution that is sensitive to rain and fog noise and considers environmental rain and fog noise modeling, so that the model is sensitive to rain and fog noise, effectively improves the quality of sensor fusion imaging, improves the robustness of the algorithm, and reduces the computing resource overhead of the sensor fusion imaging method. Specifically, the invention includes: 1) constructing a set of data acquisition equipment based on infrared light and visible light cameras to obtain paired infrared images and visible light images in different weather conditions in the same scene; 2) enhancing infrared images and visible light images, taking infrared images and visible light images obscured by rain and fog as input, and constructing a deep neural network fusion based on the adversarial learning method to generate visible light images with clear texture and no obstruction; 3) constructing a multi-sensor feature selection module based on the deep neural network, by learning the mask information corresponding to different sensor information, adaptively selecting important features for fusion and suppressing the noise corresponding to each sensor, thereby improving the algorithm's robustness to noise and reducing the model's computing resource requirements; the proposed scheme can be applied to vehicles, airplanes and other means of transportation, and by generating visible light images without rain and fog to assist drivers in obtaining information about the surrounding environment, thereby ensuring driving safety in rainy and foggy days.

[0073] It should be noted that the noise in the environment is not limited to rain and fog noise, but may also include other noises. In order to clearly illustrate the technical solution of the present invention, rain and fog noise is taken as an example for description in this embodiment; therefore, the protection scope of the present invention is not limited to rain and fog noise.

[0074] Example 1

[0075] like Figure 1 As shown, this embodiment provides a multi-sensor adaptive fusion imaging method based on infrared and visible light cameras, which may specifically include the following steps:

[0076] S1. Acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair.

[0077] Specifically, for the same environment, the original infrared and visible light data pairs P = {A, B} are obtained by infrared sensors and visible light sensors respectively. The infrared image A consists of M × N points {p1, p2, …, p M×N}, where each point p n Represents temperature information, that is, p n ∈[0,1]. The visible light image B consists of M×N points {p1,p2,…,p M×N}, where each point p n Represents the color information of the three channels of red, green and blue, that is, p n∈[0,256] 3 It should be noted that the infrared and visible light images are paired P = {A, B}, representing different features of the same environment.

[0078] S2. Perform data enhancement processing on the constructed data set to obtain an enhanced data set.

[0079] In order to obtain the detail mask used in the optimization process and the visible light image blocked by rain and fog, this embodiment performs enhancement based on the original data.

[0080] In some embodiments, participating Figure 2 , step S2 specifically includes the following steps:

[0081] S21. Obtain temperature details and texture details.

[0082] In this step, the enhanced data pairs are generated using the pre-trained deep neural network and the original data pairs. Given a deep neural network model f parameterized by θ θ (·), such as Unet, Vgg16, etc., this model can generate temperature detail mask C or texture detail mask D. Using the pre-trained deep neural network network f θ (·) Calculate the temperature detail mask C = f from the infrared image A θ (A), and the texture detail mask D = f is calculated from the visible light image B θ (B) is an enhanced dataset. A and B come from the original infrared and visible light data pair P = {A, B}. It should be noted that the mask consists of M×N points {p1, p2, …, p M×N}, where each point p n Indicates the intensity of information flow, that is, p n ∈[0,1].

[0083] S22, adding rain and fog noise to the visible light image.

[0084] As an optional implementation, rain and fog noise is added by using an image weighting method, and a predefined rain and fog image is superimposed on the visible light image to obtain a visible light image E blocked by rain and fog.

[0085] It should also be noted that the enhanced data pair is an infrared image, a visible light image, a temperature detail mask and a texture detail mask, and the visible light image data pair P = {A, B} is enhanced by the infrared and visible light data pair P = {A, B}. A ={A,B,C,D,E}.

[0086] S3. Construct an image generation model and use the enhanced data set to train the image generation model; the image generation model takes the infrared image and the obstructed visible light image as input, and outputs an unobstructed visible light image.

[0087] Specifically, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning mask information corresponding to different sensors, and suppress the noise corresponding to each sensor.

[0088] In some embodiments, the generator and the discriminator are constructed by constructing a multi-layer convolutional network. Given a generator and a discriminator defined by K convolutional layers, an image and a discrimination result can be generated. The generator Gen generates an image, and the discriminator Dis T Discriminator Dis D Discriminate the texture details of the image generated by the generator. Optionally, initialize the generator Gen and the discriminator model Dis T and Dis D The parameters of are implemented through the Xavier initialization method, where the Xavier initialization method is a popular technique for initializing weights in neural networks, which considers the number of input and output units in each layer to determine the scale of random initialization.

[0089] In order to select important features of different sensor data and suppress the noise corresponding to each sensor, this embodiment takes into account the large modal differences of sensor data and the susceptibility to environmental influences, and introduces a multi-sensor feature selection module S based on the generator Gen to handle the environmental rain and fog noise problem. The multi-sensor feature selection module S is composed of convolutional layers.

[0090] In order to enable the image generation model to learn the temperature details of infrared images and the texture details of visible light images, this embodiment optimizes the generator and the discriminator based on the generative adversarial optimization method. The working method and optimization method of each module are described in detail below.

[0091] (1) Generator Imaging

[0092] join Figure 3 First, the enhanced data pair P is input to the multi-sensor feature selection module S A = {A, B, C, D, E} infrared image A and visible light image E blocked by rain and fog, generate infrared image mask M A and visible light image mask M EThe mask operation is based on the mask, and the image is element-by-element dot multiplication is performed. Its significance is to weaken the information strength of the distorted points of the image. Then, based on the mask generated above, the infrared image A and the visible light image E blocked by rain and fog are masked to generate A m and E m :

[0093] A m =A⊙M A ,

[0094] E m =E⊙M E ,

[0095] Where ⊙ represents the matrix Hadamard product.

[0096] Finally, the infrared image A after mask operation is input to the generator Gen m and the visible light image E blocked by rain and fog m , generating a fused image G.

[0097] (2) Discriminator determines image quality

[0098] join Figure 4 First, according to the enhanced data, P A = image mask C, D of {A, B, C, D, E}, masking operation is performed on infrared image A, visible light image B and fused image G to generate masked image A C ,B D ,G C ,G D :

[0099] A C =A⊙C,

[0100] B D =B⊙D,

[0101] G C =G⊙C,

[0102] G D =G⊙D,

[0103] Then the discriminator Dis T Input image A C and image G C , get the discrimination probability T real ,T fake ; To the discriminator Dis D Input image B D and image G D , get the discrimination probability D real ,D faleAmong them, the discriminant probability R∈[0,1] represents the probability of authenticity of the input. The adversarial loss is calculated based on the discriminant probability

[0104]

[0105] (3) Loss calculation

[0106] The losses are calculated based on the structural similarity loss function and the adversarial loss function to optimize the generator and discriminator.

[0107] Specifically, according to the adversarial loss generated above Then, the structural similarity loss SSIM(A,G) and SSIM(B,G) of the fused image G, infrared image A, and visible light image B are calculated. The loss values ​​calculated above are combined into the generator loss value

[0108]

[0109] In the formula, Represents the structural similarity loss:

[0110] SSIM(x,y)=g(x,y)×h(x,y) -1 ,

[0111] g(x,y)=(2μ x μ y +C1)×(2σ x σ y +C2),

[0112]

[0113] Then, according to the adversarial loss Calculate the discriminator loss value separately

[0114]

[0115] Finally, the generator, multi-sensor feature selection module and discriminator are optimized based on the above loss values.

[0116] (4) Multi-sensor feature selection module

[0117] According to the infrared and visible light images, data is input into the multi-sensor feature selection module, and the images are dynamically fused based on the output of the multi-sensor feature selection module.

[0118] Specifically, the infrared image and the visible light image of the sensor are obtained, and the infrared image and the visible light image form a pair. The infrared image and the visible light image are input to the multi-sensor feature selection module S, and the multi-sensor feature selection module S calculates the infrared image mask M A and visible light image mask M B The average intensity produces the selectivity coefficient S of the infrared image A and the selectivity coefficient S of the visible light image B :

[0119] S A =avg(M A ),

[0120] S B =avg(M B ),

[0121] Where avg(M) represents the mean value of the information flow intensity calculated for all data points of mask M.

[0122] If and only if the selectivity coefficient S of the infrared image A Greater than the selectivity coefficient S of the visible light image B When , the infrared and visible light images are input to the generator Gen to generate the fused image G, otherwise the visible light image B is directly output. It should be noted that if and only if the selection coefficient S of the infrared image is A Greater than the selectivity coefficient S of the visible light image B When , the quality of the fused image G is higher than that of the original visible light image B. It should also be noted that the dynamic fusion of images significantly reduces the computational overhead.

[0123] In summary, the enhancement method proposed in the present invention is used to enhance infrared images and visible light images to obtain temperature detail masks, texture detail masks and visible light images blocked by rain and fog; secondly, the loss is calculated according to the structural similarity loss function and the adversarial loss function, and the generator and the discriminator are optimized so that the generator can fuse the temperature details of the infrared image and the texture details of the visible light image; finally, according to the multi-sensor feature selection module and the generator, the fusion algorithm is implemented with limited computing resources and real rain and fog noise scenes. Excellent adaptive fusion imaging performance. The embodiment of the present invention effectively solves the problem that the current fusion imaging algorithm is insensitive to rain and fog noise and has a large computing resource overhead. It can be widely used in large-scale sensor imaging assisted driving scenarios, such as vehicle and aircraft driving, etc., improves the method's anti-interference ability to rain and fog noise, breaks the current method's strong dependence on computing resources, and ensures the safety of driving in rain and fog.

[0124] Example 2

[0125] Based on the same inventive concept, an embodiment of the present invention further provides a rainy and foggy weather adaptive fusion image device based on the fusion of infrared and visible light cameras, which includes:

[0126] A data acquisition module, used to acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair;

[0127] A data enhancement module is used to perform data enhancement processing on the constructed data set to obtain an enhanced data set;

[0128] A model training module is used to construct an image generation model and train the image generation model using the enhanced data set; the image generation model takes the infrared image and the obscured visible light image as input and outputs an unobstructed visible light image;

[0129] Among them, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning mask information corresponding to different sensor information, and suppress the noise corresponding to each sensor, thereby improving the algorithm's robustness to noise.

[0130] Since the device is a device corresponding to the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras in an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0131] Example 3

[0132] See also Figure 5 Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras as described above.

[0133] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0134] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0135] Since the electronic device is an electronic device corresponding to the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0136] Example 4

[0137] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras as described above.

[0138] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0139] Since the storage medium is the storage medium corresponding to the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0140] Example 5

[0141] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the multi-sensor adaptive fusion imaging method based on infrared and visible light cameras according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0142] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0143] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0144] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-sensor adaptive fusion imaging method, characterized in that: The following steps are involved: Acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair; Perform data enhancement processing on the constructed data set to obtain an enhanced data set; An image generation model is constructed and trained using the enhanced data set; the image generation model takes the infrared image and the obstructed visible light image as input and outputs an unobstructed visible light image; Among them, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning the mask information corresponding to different sensors, and suppress the noise corresponding to each sensor.

2. The multi-sensor adaptive fusion imaging method according to claim 1, characterized in that: The step of acquiring an infrared image and a visible light image corresponding to the infrared image and constructing an image pair includes: For the same environment, an infrared image is acquired through an infrared sensor, and a visible light image is acquired through a visible light sensor, and an image pair P = {A, B} is constructed to represent different features of the same environment; The infrared image A consists of M×N points {p1, p2, ..., p M×N }, each point p i Represents temperature information; the visible light image B consists of M×N points {p1, p2, ..., p M×N }, each point p i Represents the color information of the three channels of red, green and blue.

3. The multi-sensor adaptive fusion imaging method according to claim 1, characterized in that: The step of performing data enhancement processing on the constructed data set to obtain an enhanced data set includes: Use a pre-trained deep neural network to calculate the temperature detail mask C from the infrared image A and the texture detail mask D from the visible light image B to enhance the dataset; Adding preset noise to the visible light image using a preset noise image to obtain an obscured visible light image E; wherein the preset noise includes rain and fog noise; Among them, the enhanced image pair is represented as P A ={A,B,C,D,E}.

4. The multi-sensor adaptive fusion imaging method according to claim 1, characterized in that: The generator works as follows: Input the enhanced image pair P to the multi-sensor feature selection module S A ={A, B, C, D, E} infrared image A and the obscured visible light image E, generate infrared image mask M A and visible light image mask M E ; According to the generated mask, the infrared image A and the obscured visible light image E are masked to generate image A. m and image E m : A m =A⊙M A , AND m =E ⊙ME, Where ⊙ represents the matrix Hadamard product; Input the masked image A to the generator Gen m and image E m , generate the fused image G.

5. The multi-sensor adaptive fusion imaging method according to claim 1, characterized in that: The discriminator works as follows: According to the enhanced image P A = {A, B, C, D, E}, perform mask operations on the infrared image A, the visible light image B and the fused image G to generate the masked image A C , B D , G C , GD: A C =A⊙C, B D =B ⊙ D, G C =G ⊙ C, GD=G⊙D, To the discriminator Dis T Input image A C and image G C , get the discrimination probability T real , T fake ; To the discriminator Dis D Input image B D and image G D , get the discrimination probability D real , D fale ; Calculate the adversarial loss based on the obtained discriminant probability 6. The multi-sensor adaptive fusion imaging method according to claim 5, characterized in that: The step of training the image generation model includes the steps of optimizing the generator and the discriminator: Based on the obtained adversarial loss Then, according to the structural similarity loss SSIM(A, G) and SSIM(B, G) of the fused image G and the infrared image A, and the fused image G and the visible light image B, the generator loss value is calculated. In the formula, SSIM(,) represents the structural similarity loss; According to the adversarial loss Calculate the discriminator loss value separately The generator, multi-sensor feature selection module, and discriminator are optimized according to the obtained loss values.

7. The multi-sensor adaptive fusion imaging method according to claim 1, characterized in that: The multi-sensor feature selection module works as follows: Acquire an infrared image and a visible light image acquired by a sensor, wherein the infrared image and the visible light image form an image pair; Input the infrared image and visible light image to the multi-sensor feature selection module S, and calculate the infrared image mask M respectively. A and visible light image mask M B The average intensity produces the selectivity coefficient S of the infrared image A and the selectivity coefficient S of the visible light image B : S A =avg(M A ), S B =avg(M B ), Where avg(M) represents the average value of the information flow intensity calculated for all data points of mask M; If and only if the selectivity coefficient S of the infrared image A Greater than the selectivity coefficient S of the visible light image B When , the infrared and visible light images are input to the generator Gen to generate the fused image G, otherwise the visible light image B is directly output.

8. A multi-sensor adaptive fusion imaging device, characterized in that: include: A data acquisition module, used to acquire an infrared image and a visible light image corresponding to the infrared image, construct an image pair, and construct a data set based on the image pair; A data enhancement module is used to perform data enhancement processing on the constructed data set to obtain an enhanced data set; A model training module is used to construct an image generation model and train the image generation model using the enhanced data set; the image generation model takes the infrared image and the obscured visible light image as input and outputs an unobstructed visible light image; Among them, the image generation model includes a generator, a discriminator and a multi-sensor feature selection module; the generator is used to generate a fused image; the discriminator is used to judge the quality of the generated fused image to calculate the loss; the multi-sensor feature selection module is used to adaptively select features for fusion by learning mask information corresponding to different sensor information, and suppress the noise corresponding to each sensor.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-sensor adaptive fusion imaging method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images