Environmentally adaptive dual light fusion method, apparatus, and medium

Abstract

The application provides an environment-adaptive dual-light fusion method, device and medium. The environment-adaptive dual-light fusion method obtains infrared images and visible light images of the same target scene, obtains environment information representing the target scene environment, obtains fusion weights of the infrared images and the visible light images according to the environment information, and fuses the infrared images and the visible light images according to the fusion weights to obtain dual-light fusion images. Therefore, the dual-light fusion method provided by the application can obtain dual-light fusion images with good fusion effect under different target scene environments, has high environment adaptability, can optimize the detection rate and false detection rate of target detection, is beneficial to improving the sensing performance of a target sensing module in various environment scenes, and is more convenient for human eye observation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an environment-adaptive dual-light fusion method, apparatus, and medium. Background Technology

[0002] Object detection is a crucial function in driver assistance and autonomous driving systems. Modules such as forward collision warning and pedestrian collision warning rely heavily on accurate object detection. However, real-world scenarios present various weather conditions. The risk of accidents is more than double that under normal conditions at night and in rainy weather, while fog, haze, sandstorms, and strong sunlight severely reduce visibility in visible light, posing significant challenges to object detection.

[0003] LiDAR has garnered significant attention due to its ability to accurately perceive the surrounding three-dimensional environment, with detection precision within centimeters, and its insensitivity to lighting conditions. However, it suffers from severe interference, even malfunctioning, when encountering obstacles, rain, fog, or sandstorms. Millimeter-wave radar, on the other hand, is objectively better suited to humid weather, but its low spatial resolution limits its target classification capabilities. Infrared cameras detect the thermal radiation of objects, are unaffected by lighting conditions, have even greater detection ranges at night, can penetrate dense fog, and possess high resolution. Therefore, dual-light fusion technology, which combines the advantages of infrared and visible light images, has emerged.

[0004] However, target detection based on existing dual-light fusion technology still has certain deviations and low accuracy in different application environments, which is not conducive to further improving the safety performance of assisted driving. Summary of the Invention

[0005] To at least partially improve the technical problems existing in the prior art, this application provides a dual-light fusion method, device and computer-readable medium that can improve the accuracy of target detection to enhance the safety performance of assisted driving and adapt to the environment.

[0006] According to a first aspect of the embodiments of this application, an environment-adaptive dual-light fusion method is provided, comprising:

[0007] Acquire infrared and visible light images of the same target scene within the same group;

[0008] Obtain environmental information characterizing the target scene environment, and obtain the fusion weights of the infrared image and the visible light image based on the environmental information;

[0009] The infrared image and the visible light image are fused according to the fusion weight to obtain a dual-light fused image.

[0010] According to a second aspect of the embodiments of this application, an environment-adaptive dual-light fusion device is provided, including a memory and a processor. The memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the dual-light fusion method.

[0011] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the dual-light fusion method described above.

[0012] As can be seen from the above, the environment-adaptive dual-light fusion method provided in this application acquires infrared and visible light images of the same target scene, as well as environmental information characterizing the target scene environment. Based on the environmental information, it obtains the fusion weights of the infrared and visible light images, and then fuses them according to these fusion weights to obtain a dual-light fused image. The dual-light fusion method provided in this application can achieve good fusion results by performing dual-light fusion of infrared and visible light images according to different fusion weights based on different environmental information in different target scene environments. Therefore, compared with existing dual-light fusion methods, the environment-adaptive dual-light fusion method provided in this application has higher environmental adaptability, can optimize the detection rate and false detection rate of target detection, is beneficial to improving the perception performance of the target perception module in various environmental scenarios, and is also easier for the human eye to observe. Attached Figure Description

[0013] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0014] Figure 1 A schematic diagram of the structure of an optional environment-adaptive dual-light fusion device for the environment-adaptive dual-light fusion method provided in the embodiments of this application;

[0015] Figure 2 This is a schematic flowchart of an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0016] Figure 3 This is a schematic flowchart of a method for obtaining environment-adaptive fusion weights in a dual-light fusion method provided according to some embodiments of this application;

[0017] Figure 4 This is a schematic diagram illustrating the steps of performing two-light fusion based on fusion weights in an environment-adaptive two-light fusion method provided according to some embodiments of this application;

[0018] Figure 5This is a schematic diagram illustrating the steps of registering infrared target detection results and visible light target detection results in an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0019] Figure 6 This is a schematic diagram illustrating the registration and calibration steps based on texture information in an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0020] Figure 7 This is a schematic diagram illustrating the effect of fusing infrared and visible light images to obtain a dual-light fused image in a nighttime environment using an environment-adaptive dual-light fusion method provided in some embodiments of this application.

[0021] Figure 8 This is a schematic diagram illustrating the effect of fusing infrared and visible light images to obtain a dual-light fused image in a rainy environment using an environment-adaptive dual-light fusion method provided in some embodiments of this application.

[0022] Figure 9 This is a schematic diagram illustrating the effect of fusing infrared and visible light images to obtain a dual-light fused image in a visually obstructed environment using an environment-adaptive dual-light fusion method provided in some embodiments of this application.

[0023] Figure 10 This is a schematic diagram of an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0024] Figure 11 This is a schematic diagram of the process of registering an infrared image with a visible light image in an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0025] Figure 12 This is a schematic diagram illustrating the process of calibrating registration based on target texture information in an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0026] Figure 13 This is a flowchart illustrating the process of obtaining the environment category of a target scene environment in an environment-adaptive dual-light fusion method provided according to some embodiments of this application;

[0027] Figure 14 This is a schematic diagram of the structure of an environment-adaptive dual-light fusion device provided according to some embodiments of this application. Detailed Implementation

[0028] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the ways in which this application may be implemented. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0030] In the following description, the expression “some embodiments” is used, which describes a subset of possible embodiments. However, it should be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0031] Please see Figure 1The diagram shown is a schematic representation of an optional environment-adaptive dual-light fusion device for the environment-adaptive dual-light fusion method provided in this application embodiment. The environment-adaptive dual-light fusion device includes a processor 11, a memory 12 connected to the processor 11, and further includes an infrared camera 13 and a visible light camera 14 connected to the processor 11. The infrared camera 13 and the visible light camera 14 are fixed in the same structure. In some embodiments, the optical axes of the infrared camera 13 and the visible light camera 14 are parallel. When acquiring image data of a target object, the infrared camera 13 and the visible light camera 14 are clock-synchronized, simultaneously acquiring infrared and visible light images of the target object from the same target scene. The infrared and visible light images of the same target scene acquired at the same time are considered as a group, referred to as the same group of infrared and visible light images. An environment-adaptive dual-light fusion device acquires infrared and visible light images of the same group of objects in real time using an infrared camera 13 and a visible light camera 14, and sends them to a processor 11. A memory 12 stores a computer program implementing the adaptive dual-light fusion method provided in this application embodiment. The processor 11 executes the computer program to obtain the fusion weights of the infrared and visible light images based on environmental information characterizing the target scene environment, and then fuses the infrared and visible light images according to the fusion weights to obtain a dual-light fused image. The environment-adaptive dual-light fusion device can be various intelligent terminals integrating an infrared camera 13 and a visible light camera 14, and possessing storage and processing functions, such as handheld observation devices, various aiming devices, security monitoring equipment, vehicle-mounted / airborne equipment, etc. Further, in some embodiments, the environment-adaptive dual-light fusion device further includes an environmental sensor 15, which acquires environmental information characterizing the target scene environment and sends the environmental information to the processor 11. Further, the environmental sensor 15 includes, but is not limited to, an ambient light sensor 151 for acquiring ambient light information of the target scene environment and an ambient humidity sensor 152 for acquiring ambient humidity information of the target scene environment.

[0032] Please see Figure 2 The diagram shown is a schematic flowchart of an environment-adaptive dual-light fusion method according to some embodiments of this application. The dual-light fusion method provided in the embodiments of this application includes steps S02, S04, and S06, and the specific description of each step is as follows.

[0033] S02: Acquire infrared and visible light images of the same target scene.

[0034] The same set of infrared and visible light images refers to infrared and visible light images acquired simultaneously from the same target scene. The target scene can be any scene where images are acquired using an environment-adaptive dual-light fusion device. The environment-adaptive dual-light fusion device may include a shooting module, which includes an infrared camera and a visible light camera. Acquiring the same set of infrared and visible light images for the target scene includes: the environment-adaptive dual-light fusion device acquiring infrared and visible light images of the target scene in real time through the shooting module. In some alternative embodiments, the environment-adaptive dual-light fusion device does not include a shooting module, and acquiring the same set of infrared and visible light images for the target scene includes: the environment-adaptive dual-light fusion device acquiring the same set of infrared and visible light images sent by other intelligent devices with image capturing capabilities, where other intelligent devices can be mobile terminals, the cloud, etc.

[0035] In some embodiments, the center of the lens of the infrared camera for acquiring infrared images is close to the center of the lens of the visible light camera for acquiring visible light images, that is, the optical axes of the infrared camera and the visible light camera are parallel and the distance between them is small. Preferably, the infrared camera and the visible light camera are on the same optical axis, so as to obtain a mapping matrix between the infrared image and the visible image.

[0036] S04: Obtain environmental information representing the target scene environment, and obtain the fusion weights of infrared and visible light images based on the environmental information.

[0037] The target scene environment refers to the environment in which the target is located, which is included in both infrared and visible light images. The target is the object that the user needs to focus on, i.e., the object that the aforementioned shooting modules are aimed at. The aforementioned shooting modules include infrared cameras and visible light cameras. Environmental information includes, but is not limited to, light intensity information, humidity information, and weather information in the target scene.

[0038] Fusion weight refers to the infrared weight coefficient multiplied by the corresponding pixel in the infrared image and the visible light weight coefficient multiplied by the corresponding pixel in the visible light image when fusing infrared and visible light images. The sum of the infrared weight coefficient and the visible light weight coefficient is 1.

[0039] S06: According to the fusion weight, the infrared image and the visible light image are fused to obtain a dual-light fused image.

[0040] The fusion weights for fusing infrared and visible light images are determined based on environmental information. Different environmental information in the target scene results in different fusion weights, leading to different fused images obtained by fusing the infrared and visible light images according to these weights. The fused image exhibits good fusion performance across various target scene environments. The environmental fused image includes target information from both the infrared and visible light images. The degree to which the target information in the infrared and visible light images influences the target information in the fused image is determined by the aforementioned fusion weights.

[0041] As can be seen from the above, the environment-adaptive dual-light fusion method provided in this application acquires infrared and visible light images of the same target scene, as well as environmental information characterizing the target scene environment. Based on the environmental information, it obtains the fusion weights of the infrared and visible light images, and then fuses them according to these weights to obtain a dual-light fused image. The dual-light fusion method provided in this application can achieve good fusion results in different target scene environments by performing dual-light fusion of infrared and visible light images according to different fusion weights based on different environmental information. Therefore, compared with existing dual-light fusion methods based on image brightness adaptation, the environment-adaptive dual-light fusion method provided in this application has higher environmental adaptability, can optimize the detection rate and false detection rate of target detection, is beneficial to improving the perception performance of the target perception module in various environmental scenarios, and is also easier for the human eye to observe.

[0042] Please see Figure 3 The diagram shown is a flowchart illustrating a method for obtaining environment-adaptive fusion weights in a dual-light fusion method according to some embodiments of this application. In this embodiment, S04 further includes S042, S044, and S046, and the steps are described below.

[0043] S042: Obtain the ambient light and humidity information of the target scene environment respectively.

[0044] Specifically, ambient light and humidity information can be acquired using ambient light and humidity sensors, respectively. Ambient light information characterizes the light intensity of the target scene environment. Based on this information, it can be determined whether the infrared and visible light images were acquired during the day when light intensity is high or at night when light intensity is low. Ambient humidity information characterizes the humidity level of the target scene environment. Based on this information, it can be determined whether the weather in the target scene environment was rainy or sunny when the infrared and visible light images were acquired.

[0045] S044: Based on feature extraction and recognition of infrared and / or visible light images, determine whether the weather in the target scene is visibility-impairing weather and obtain the determination result.

[0046] A trained neural network that can learn visibility-impairing weather features is used to extract and identify features from infrared and / or visible light images to determine whether the weather in the target scene is visibility-impairing.

[0047] Visibility-impairing weather refers to a type of weather where the air becomes turbid due to the presence of water vapor condensation, dry suspended matter, etc., resulting in reduced visibility. Visibility-impairing weather includes fog, light fog, snowstorms, blowing snow, sandstorms, blowing dust, dust, smoke, and haze.

[0048] S046: Based on ambient light information, ambient humidity information, and judgment results, determine the environmental category of the target scene environment, and obtain the fusion weight of the infrared image and the visible light image based on the environmental category of the target scene environment.

[0049] In some embodiments, the environmental information characterizing the target scene environment includes ambient light information, ambient humidity information, and a judgment result. Different environmental information corresponds to different environmental categories of the target scene environment, and different environmental categories correspond to different fusion weights. Therefore, the dual-light fusion method provided in this application adapts to changes in the environment, selecting fusion weights corresponding to the target scene environment to fuse infrared and visible light images, achieving good fusion results in different environments.

[0050] Furthermore, in some embodiments, S046 specifically includes: if the determination result in S044 indicates that the weather of the target scene is visibility-impairing weather, then the environmental category of the target scene is determined to be a visibility-impairing environment; if the determination result indicates that the weather of the target scene is non-visibility-impairing weather, then the environmental category of the target scene environment is determined based on ambient light information and ambient humidity information. The environmental category includes at least one of daytime rainy weather, daytime sunny weather, nighttime rainy weather, and nighttime sunny weather.

[0051] In some embodiments, light intensity ranges corresponding to daytime and nighttime environments can be set separately, as well as humidity ranges corresponding to sunny and rainy environments. Based on the light intensity and humidity information of the target scene environment, the light intensity and humidity ranges within which the target scene environment falls can be determined, thereby identifying the environment category corresponding to the target scene environment. Specifically, the ambient light information determines whether the target scene environment is a daytime or nighttime environment, and the ambient humidity information determines whether the target scene environment is a rainy or sunny environment.

[0052] In the dual-light fusion method provided according to some embodiments of this application, the environmental category of the target scene environment is divided into five types: daytime rainy environment, daytime sunny environment, nighttime sunny environment, and nighttime rainy environment under non-visual-range obstacles, as well as visual-range obstacle environment. In other embodiments, the method of classifying environmental categories is not limited to this.

[0053] Furthermore, in some embodiments, S04 may specifically include: obtaining an environment category selection instruction as environment information, and determining the fusion weight of the infrared image and the visible light image based on the environment category corresponding to the environment category selection instruction. In some embodiments, the environment category includes at least one of nighttime clear weather environment, nighttime rainy weather environment, daytime clear weather environment, daytime rainy weather environment, and visibility-impaired environment. Moreover, the order in which the three steps of obtaining the judgment result in S044, determining whether the target scene environment is a daytime environment or a nighttime environment based on the ambient light information, and determining whether the target scene environment is a rainy environment or a clear weather environment based on the ambient humidity information are not limited.

[0054] In some embodiments, after determining the environment category of the target scene environment, determining the fusion weight based on the environment category of the target scene environment specifically includes: obtaining the fusion weight of the infrared image and the visible light image based on the environment category of the target scene environment and a preset mapping relationship between environment category and fusion weight.

[0055] In some embodiments, different environment categories that the target scene may belong to are preset, and a mapping relationship between each environment category and the fusion weight is preset. Therefore, after obtaining the environment category of the target scene environment, a fusion weight matching the target scene environment can be obtained based on the environment category and the preset mapping relationship.

[0056] The preset environment category and fusion weight mapping relationship includes a preset environment category and fusion weight mapping relationship table. Based on the environment category of the target scene environment and the preset environment category and fusion weight mapping relationship, the fusion weight of the infrared image and the visible light image is obtained. Specifically, based on the environment category of the target scene environment, the fusion weight corresponding to the target scene is directly obtained by searching the preset environment category and fusion weight mapping relationship table.

[0057] The preset mapping relationship between environment categories and fusion weights can further include a preset calculation formula between environment categories and fusion weights. For example, in some embodiments, if the environment category of the target scene is a nighttime rainy environment, then the fusion weight corresponding to the nighttime environment and the fusion weight corresponding to the rainy environment are determined according to the mapping relationship table between environment categories and fusion weights. Then, the fusion weight corresponding to the nighttime rainy environment is calculated according to the fusion weight corresponding to the nighttime environment, the fusion weight corresponding to the rainy environment, and the preset calculation formula.

[0058] Clear night A 1-A Rainy night B 1-B Sunny during the day C 1-C Rainy day D 1-D Foggy weather E 1-E

[0059] Table 1

[0060] In some embodiments, the preset environment category and fusion weight mapping table is shown in Table 1, where A, B, C, D, and E are weight values ​​between 0 and 1.0. In this embodiment, the field-of-view obstacle environment refers to foggy weather.

[0061] As can be seen from the above, in the dual-light fusion method provided according to some embodiments of this application, ambient light information and ambient humidity information of the target scene environment can be obtained based on ambient light sensors and ambient humidity sensors, respectively. Then, based on the ambient light information, it can be determined whether the target scene environment is currently in daytime or nighttime, and based on the ambient humidity information, it can be determined whether the target scene environment is currently in rainy or sunny weather. Furthermore, by combining the recognition results of infrared and visible light images, it can be determined whether the weather of the target scene is foggy or other severe visibility-impairing weather. Based on the results of each determination, the current environment category of the target scene environment is determined. Then, according to the environment category to which the target scene environment belongs, the corresponding fusion weight is selected to achieve automatic environment-adaptive dual-light fusion. In other embodiments, the user can also manually input the corresponding environment category selection command into the processor of the environment-adaptive dual-light fusion device based on physical buttons or virtual buttons in the interface, as the environmental information representing the target scene environment. The environment category selected according to the environment selection command is used as the environment category of the target scene environment, thus achieving manual environment-adaptive dual-light fusion.

[0062] Common dual-light fusion detection methods based on the fusion of infrared and visible light images include pixel-level dual-light fusion detection, feature-level fusion detection, and decision-level fusion detection. Pixel-level dual-light fusion refers to a fusion method that first fuses the pixels of the infrared and visible light images before performing target detection. Feature-level fusion refers to a fusion method that first fuses target features from a single modality before performing target detection. For feature-level fusion, deep learning networks have been widely used in recent years to extract multi-layer convolutional features to achieve target fusion detection from visible light and infrared images. However, these algorithms require a large amount of data for accurate registration, and differences in feature types often lead to excessive fusion difficulty and computational cost, severely limiting the real-time performance of detection tasks. Decision-level fusion detection refers to performing target detection on multi-modal data (such as infrared and visible light images) separately, and then fusing the results; it is a more advanced form of information fusion.

[0063] Based on the above description, in some embodiments, this application provides a method for fusing infrared and visible light images based on decision-level fusion detection. Please refer to the following for details. Figure 4The diagram illustrates the steps of performing two-light fusion based on fusion weights in an environment-adaptive two-light fusion method according to some embodiments of this application. In this embodiment, S06 specifically includes S062, S064, and S066, and the descriptions of each step are as follows.

[0064] S062: Perform target detection on the infrared image and the visible light image respectively, and obtain the infrared target detection result and the visible light target detection result.

[0065] S064: Register the infrared target detection results and the visible light target detection results to obtain the registration result.

[0066] S066: Based on the registration results, the infrared target detection results and visible light target detection results registered are fused according to the fusion weights to obtain a dual-light fused image carrying the target detection results.

[0067] Target detection is performed on infrared images using target detection algorithms to obtain infrared target detection results. The same algorithm is also used to perform target detection on visible light images to obtain visible light target detection results. The infrared target detection results include the infrared target detection bounding box information that encloses the target in the infrared image during detection. The visible light target detection results include the visible light target detection bounding box information that encloses the target in the visible light image during detection. The target detection bounding box information further includes the side length and center point position information of the corresponding target box.

[0068] In some embodiments, an infrared image target detection network is established based on a deep convolutional network method with multi-feature fusion to achieve real-time detection of targets in infrared images and obtain infrared target detection results. In addition, a visible light image target detection network is established based on a deep convolutional network method with multi-feature fusion to achieve real-time detection of targets in visible light images and obtain visible light target detection results.

[0069] In both infrared image target detection networks and visible light image target detection networks, the backbone network used as the feature extraction network can be a network such as Darknet or ResNet. After the backbone network, a feature pyramid network can be used as the intermediate feature extraction network Neck. Finally, both networks can use a multi-level feature fusion method to perform feature fusion at different levels, and use loss functions such as IOU loss and focal loss to train their respective target detection network models.

[0070] Registration in S064 is also called alignment. Registering infrared target detection results and visible light target detection results means aligning the infrared target detection bounding boxes and the visible light target detection boxes. The registration results are aligned within the same scene to ensure pixel-level correspondence. The registration result refers to the registered infrared detection results and visible light detection results.

[0071] A two-light fusion image carrying target detection results refers to a two-light fusion image labeled with target detection bounding box information and target category information. A two-light fusion method that fuses the registered infrared and visible light detection results according to adaptive fusion weights based on changes in the target scene environment is an environment-adaptive and decision-level-based fusion method. It exhibits good fusion results in different target scene environments, facilitating improvements in target detection accuracy.

[0072] Before fusing infrared and visible light images, they need to be registered. One implementation for registering infrared and visible light images is shown in S064, which achieves registration by registering the infrared target detection results and the visible light target detection results. In other embodiments, registering infrared and visible light images may also include: finding local feature points (pixels) in the same group of infrared and visible light images in the same target scene, then calculating the mapping matrix between the infrared and visible light images using the least squares method, and then registering the infrared and visible light images based on this mapping matrix. However, in this method, calculating the mapping matrix requires calibrating the infrared and visible light cameras, and determining the mapping matrix based on the calibration results, making the method relatively complex.

[0073] To facilitate obtaining the mapping matrix between infrared and visible light images, some embodiments employ a fixed optical axis relationship between a dual-light camera system (an infrared camera for acquiring infrared images and a visible light camera for acquiring visible light images). This ensures that the optical centers of the infrared and visible light cameras are as close as possible, meaning their optical axes are parallel and closely spaced. This allows the mapping matrix to be obtained directly analytically without the need for dual-camera calibration, simplifying the implementation of environment-adaptive dual-light fusion methods. By setting the optical axes of the infrared and visible light cameras to be parallel and closely spaced, or even coaxial, the mapping matrix between the infrared and visible light images can be obtained using the intrinsic parameter information of the infrared and visible light cameras.

[0074] Please see Figure 5The diagram illustrates the steps of registering infrared target detection results and visible light target detection results in an environment-adaptive dual-light fusion method provided according to some embodiments of this application. In this embodiment, the optical axes of the infrared camera used to acquire infrared images and the visible light camera used to acquire visible light images are parallel, and S064 specifically includes S0642, S0644, and S0646.

[0075] S0642: Based on the infrared target detection results and the optical axis information of the infrared camera, determine the infrared target angle relative to the center of the infrared camera's optical axis in the infrared image.

[0076] S0644: Based on the visible light target detection results and the optical axis information of the visible light camera, determine the visible light target angle in the visible light image relative to the center of the visible light camera's optical axis.

[0077] S0646: Match the infrared target angle and the visible light target angle. When the infrared target angle and the visible light target angle meet the preset matching conditions, determine the corresponding infrared target detection result and the visible light target detection result for registration and obtain the registration result.

[0078] The order in which S0642 and S0644 are executed is not limited in this application; they can be executed sequentially or simultaneously. The infrared target angle refers to the angle between the line connecting the center point of the infrared target detection frame to the center of the infrared camera lens and the optical axis of the infrared camera. The visible light target angle refers to the angle between the line connecting the center point of the visible light target detection frame to the center of the visible light camera lens and the optical axis of the visible light camera.

[0079] Matching the infrared target angle with the visible light target angle involves comparing the infrared target angle with the visible light target angle separately to see if the angular deviation between the two meets a preset deviation condition. If it does, it means that the infrared target angle and the visible light target angle are matched, indicating that the infrared detection result corresponding to the matched infrared target angle and the visible light target detection result corresponding to the visible light target angle are paired, and vice versa. In this embodiment, registration is performed based on the matching of the infrared target angle and the visible light target angle. The registration method is simple and can be quickly achieved.

[0080] In some embodiments, the infrared target angle is used This indicates the visible light target angle VL_obj angle The infrared target angle is matched with the visible light target angle. When the absolute value of the angle deviation between the two is less than the threshold thres, the corresponding infrared detection result and visible light detection result are considered to be successfully registered. Specifically, the configuration formula for registration based on target angle matching is as follows: when the matching result is 1, it indicates that the registration is successful, and when it is 0, it indicates that the registration is unsuccessful.

[0081]

[0082] Registering infrared target detection results with visible light target detection results means that the absolute value of the angular deviation between the infrared target angle corresponding to the registered infrared target detection result and the visible light target angle corresponding to the registered visible light target detection result is less than the threshold thres.

[0083] In some embodiments of the dual-light fusion method provided in this application, the infrared target detection results and the visible light target detection results are registered based on the matching between the infrared target angle and the visible light target angle. In other embodiments, the infrared target detection results and the visible light target detection results can also be registered based on the mapping matrix between the infrared image and the visible light image. In this case, S064 may specifically include: registering the infrared target detection results and the visible light target detection results according to the mapping matrix between the infrared image and the visible light image determined based on the intrinsic parameter information of the infrared camera and the visible light camera, to obtain a registration result.

[0084] In some embodiments, since the optical axes of the infrared camera and the visible light camera are set to be parallel and the optical axis spacing is small, the mapping matrix between the infrared image and the visible light image can be roughly calculated directly based on the intrinsic parameters of the infrared camera and the visible light camera, and then registration can be performed based on the mapping matrix.

[0085] In some embodiments of the dual-light fusion method provided in this application, cases where the optical axes of the infrared camera and the visible light camera are parallel and the distance between them is small are equated to cases where the infrared camera and the visible light camera are dual-light cameras with the same optical axis. This facilitates the acquisition of the mapping matrix between the infrared image and the visible light image. However, the mapping matrix obtained in this way has certain errors. Therefore, the registration results of the infrared target detection results and the visible light target detection results based on target angle matching or based on the mapping matrix are coarse registration results.

[0086] To further improve the registration accuracy of infrared target detection results and visible light target detection results, some embodiments of the dual-light fusion method provided in this application further include calibrating the coarse registration result obtained in S064 above. Please refer to... Figure 6The diagram illustrates the steps of registration calibration based on texture information in an environment-adaptive dual-light fusion method provided according to some embodiments of this application. In this embodiment, S06 includes S062, S064, and S066, and further includes, between S064 and S066, calibrating the registration result in S064 based on the texture information of the target to obtain a fine-tuned result. Specifically, the step of calibrating the registration result in S064 based on the texture information of the target includes S0652 and S0654, and the descriptions of each step are as follows.

[0087] S0652: Based on the registration results, obtain the infrared target texture information and visible light target texture information corresponding to the paired infrared target detection results and visible light target detection results, respectively.

[0088] S0654: The registration results are calibrated based on the infrared target texture information and the visible light target texture information.

[0089] Infrared target texture information refers to the target texture information extracted from the infrared target detection box based on the infrared target detection results, while visible light target texture information refers to the target texture information extracted from the visible light target detection box based on the visible light target detection results.

[0090] In some embodiments, texture matching can be performed on infrared target texture information and visible light target texture information, and the registration result in S064 can be corrected based on the matching result to obtain a more accurate registration result.

[0091] Furthermore, in some embodiments, S0652 specifically includes: obtaining an infrared target region and a visible light target region with the same resolution based on the infrared target detection result, the visible light target detection result, and the mapping matrix between the infrared image and the visible light image; calculating the texture information of the infrared target region and the visible light target region respectively; and obtaining the infrared target texture information and the visible light target texture information.

[0092] Specifically, in some embodiments, the specific steps for calibrating the registration result in S064 based on the target texture information include: obtaining a visible light target detection area based on the infrared target detection result and expanding it to a certain extent; obtaining a visible light target area based on the visible light target detection result and expanding it to a certain extent; scaling the expanded infrared target detection area to the visible light target area according to the mapping matrix to obtain an infrared target area with the same size as the visible light target area; then calculating the texture information of the infrared target area and the visible light target area respectively to obtain infrared target texture information and visible light target texture information; and then calibrating the registration result in S064 based on the obtained texture information to obtain a fine registration result.

[0093] As can be seen from the above, in some embodiments according to this application, infrared and visible light cameras with parallel optical axes are used to acquire infrared and visible light images of the same scene. The infrared and visible light images are then input into corresponding deep learning networks to obtain infrared target detection results and visible light target detection results. Initial registration is performed on the infrared target detection results and visible light target detection results by matching the infrared target angle and the visible light target angle or by using the mapping matrix between the infrared and visible light images. The initial registration results are then calibrated using the texture information of the infrared target region and the visible light target region to obtain fine registration results. Furthermore, the daytime or nighttime environment and the sunny or rainy environment are determined by environmental sensors. The environmental category of the target scene is obtained by combining the infrared image and / or visible light image analysis, thereby obtaining an environment-adaptive fusion weight. Alternatively, the user can manually select the environment category and then select the corresponding fusion weight. Finally, the infrared image and visible light image are fused according to the environment-adaptive fusion weight and the calibrated fine registration results to obtain a dual-light fused image. Therefore, the dual-light fusion method provided in some embodiments of this application can efficiently fuse infrared images and visible light images in different environments, thereby optimizing the detection rate and false detection rate of target detection, improving the performance of the perception module in various environmental scenarios, and making it easier for the human eye to observe.

[0094] To further demonstrate the technical effects achieved by the environment-adaptive dual-light fusion method provided in the embodiments of this application, Figures 7 to 9 The illustrations show the effects of fusing infrared and visible light images to obtain fused images under different environments using an environment-adaptive dual-light fusion method based on some embodiments of this application. Specifically, Figure 7 This diagram illustrates the effect of fusing infrared and visible light images to obtain a fused image in a nighttime environment using an environment-adaptive dual-light fusion method provided in some embodiments of this application. Figure 8 This diagram illustrates the effect of fusing infrared and visible light images to obtain a fused image in a rainy environment using an environment-adaptive dual-light fusion method provided in some embodiments of this application. Figure 9 This diagram illustrates the effect of fusing infrared and visible light images to obtain a fused image in an environment with visual impairment, based on some embodiments of this application using an environment-adaptive dual-light fusion method. Figure 9 The visibility-impairing weather condition described is snowy. Figures 7 to 9 It can be seen that the dual-light fusion method provided by some embodiments of this application has achieved good fusion results in different environments.

[0095] Please see Figure 10The diagram shown is a schematic representation of an environment-adaptive dual-light fusion method provided according to some embodiments of this application. The dual-light fusion method provided in this application includes steps S1 to S7 after execution begins, and each step is described below.

[0096] S1: Acquire infrared and visible light images under the same field of view. The same field of view refers to the same target scene mentioned above. The specific implementation of acquiring infrared and visible light images in S1 can be found in S02, and will not be repeated here.

[0097] S2: Infrared target detection. The infrared image obtained in S1 is used to perform target detection based on an infrared image target detection network to obtain the infrared target detection results.

[0098] S3: Visible light target detection. Based on the visible light image target detection network, target detection is performed on the visible light image obtained in S1 to obtain the visible light target detection results. S2 and S3 can be executed sequentially or simultaneously.

[0099] S4: Register the infrared target with the visible light target. Specifically, register the infrared target detection results and the visible light target detection results obtained in S2 and S3 to obtain a coarse registration result.

[0100] S5: Calibrate the registration result obtained in the previous step based on the texture information. The previous step, S4, involved obtaining the texture information, which includes the infrared target texture information of the infrared target region and the visible light target texture information of the visible light target region. This step yields the calibrated, finely registered result.

[0101] S6: Obtain environmental information based on environmental sensors or user operation. Environmental sensors include, but are not limited to, ambient light sensors and ambient humidity sensors. Environmental information includes, but is not limited to, ambient light information obtained from the ambient light sensor and ambient humidity information obtained from the ambient humidity sensor. User operation specifically refers to the user inputting an environmental category selection command to the environment-adaptive dual-light fusion device, which serves as the environmental information.

[0102] S7: Perform adaptive dual-light fusion based on environmental information to obtain a dual-light fused image. For details of this step, please refer to S08 and its specific implementation in various embodiments.

[0103] Please see Figure 11 The diagram illustrates the process of registering an infrared image with a visible light image in an environment-adaptive dual-light fusion method according to some embodiments of this application. After registration begins in the dual-light fusion method provided in this application embodiment, step S4 further includes steps S41 to S6, each described below.

[0104] S41: Calculate the infrared target angle. Refer to the above description for the calculation method.

[0105] S42: Calculate the visible light target angle. Refer to the description above for the calculation method. The order in which S41 and S42 are executed is not limited; they can be executed sequentially or simultaneously.

[0106] S43: Angle matching based on the matching algorithm. The infrared target angle obtained in S41 and the visible light target angle obtained in S42 are matched based on the above formula.

[0107] S44: Determine if the absolute value of the matched angle deviation is less than the threshold threshold. If the result of S44 is yes, then execute S45; otherwise, execute S46.

[0108] S45: Marker matching successful. This means the infrared target detection result and the visible light target detection result have been successfully registered.

[0109] S46: Flag not matched. This means that the infrared target detection result and the visible light target detection result were not successfully registered.

[0110] Please see Figure 12 The diagram illustrates the process of calibrating registration based on target texture information in an environment-adaptive dual-light fusion method according to some embodiments of this application. After calibration begins in the dual-light fusion method provided in this application embodiment, step S5 further includes steps S51 to S56, each described below.

[0111] S51: Obtain the paired visible light target region. Specifically, the visible light target detection region can be directly used as the visible light target region from which texture information is to be extracted, based on the visible light target detection results.

[0112] S52: Obtain the corresponding infrared target detection area. Specifically, the infrared target detection area can be obtained based on the infrared target detection results.

[0113] S53: Scale the infrared target detection area to the visible light target area according to the mapping matrix. This step is to obtain an infrared target area with the same resolution as the visible light target area.

[0114] S54: Calculate the texture information of the visible light target region and the infrared target region. This step obtains the aforementioned visible light target texture information and infrared target texture information.

[0115] S55: Calibration using texture information. That is, calibrating the initial registration result obtained in S4 based on the texture information obtained in S54.

[0116] S56: Output calibration results. The calibration results are the fine registration results after calibrating the initial registration results based on texture information.

[0117] Please see Figure 13 The diagram illustrates the process of obtaining the environment category of a target scene environment in an environment-adaptive dual-light fusion method provided according to some embodiments of this application. After the dual-light fusion method provided in this application begins obtaining the environment category, step S7 further includes steps S71 to S74, each described below.

[0118] S71: Determines whether it is daytime or nighttime based on the ambient light sensor. Specifically, it determines whether the target scene environment is currently daytime or nighttime based on the ambient light information obtained from the ambient light sensor.

[0119] S72: Determines whether it is a rainy or sunny day based on the ambient humidity sensor. Specifically, it determines whether the target scene environment is currently rainy or sunny based on the ambient humidity information obtained from the ambient humidity sensor.

[0120] S73: Combine infrared and visible light image information to comprehensively determine whether the weather is obstructing visibility. That is, based on the recognition results of infrared and visible light images, determine whether the weather of the target scene is obstructing visibility, such as heavy fog or heavy snow, and obtain the corresponding judgment result.

[0121] S74: Output the perceived environment category. Based on the results of the judgments in S71 and S73, determine the current environment category of the target scene.

[0122] As can be seen from the above, the environment-adaptive dual-light fusion method provided according to some embodiments of this application can achieve at least one of the following technical effects.

[0123] 1. By setting the infrared camera and the visible light camera as dual-light cameras with similar optical axes (parallel and very small spacing), that is, the center of the lens of the infrared camera and the center of the lens of the visible light camera are as close as possible, the mapping matrix of the infrared image and the visible light image can be calculated through the parameters (intrinsic parameters) of the infrared camera and the visible light camera, avoiding the need to calibrate the dual-light extrinsic parameters through feature matching to obtain the mapping matrix, thus improving efficiency.

[0124] 2. The infrared target detection results and visible light target detection results obtained by deep learning networks are used to perform dual-light fusion at the decision level. The initial registration is performed using the mapping matrix or target angle matching. The image features in the target box of each pair of paired infrared and visible light targets are extracted. The texture information of the targets is used to calibrate the initial registration results, thereby improving the registration accuracy of infrared and visible light.

[0125] 3. By using environmental sensors, including but not limited to ambient light sensors and ambient humidity sensors, environmental perception results are obtained. Different fusion weights are configured according to different environmental categories. Users can also manually select the environmental category to determine the corresponding fusion weight in a manual fusion mode, which effectively improves the environmental adaptability of the fusion.

[0126] In some embodiments, this application also provides an environment-adaptive dual-light fusion device, the structural schematic of which is shown below. Figure 14 As shown. The environment-adaptive dual-light fusion device provided in this application embodiment includes an acquisition module 101, a weight determination module 102, and a fusion module 103. Specifically, the acquisition module 101 is used to acquire infrared images and visible light images of the same target scene. The weight determination module 102 is used to acquire environmental information characterizing the target scene environment and obtain the fusion weights of the infrared images and visible light images based on the environmental information. The fusion module 103 is used to fuse the infrared images and visible light images according to the fusion weights to obtain a dual-light fused image. The environment-adaptive dual-light fusion device provided in this application embodiment achieves the same technical effect as the environment-adaptive dual-light fusion method provided in this application embodiment, and will not be described again here.

[0127] Optionally, the acquisition module 101 is specifically used to acquire infrared images and visible light images in the dual-light fusion method provided in any embodiment of this application.

[0128] Optionally, the weight determination module 102 obtains the fusion weight based on the method of obtaining fusion weight according to environmental information in the dual-light fusion method provided in any embodiment of this application.

[0129] Optionally, the fusion module 103 is specifically based on the dual-light fusion method provided in any embodiment of this application, which fuses the infrared image and the visible light image according to the fusion weight to obtain a dual-light fused image.

[0130] In some embodiments, this application also provides, as Figure 1 The illustrated dual-light fusion device is an environment-adaptive device. The dual-light fusion device provided in this application includes a processor 11 and a memory 12. The memory 12 stores a computer program executable by the processor 11. When the computer program is executed by the processor 11, it implements the dual-light fusion method provided in any embodiment of this application. The environment-adaptive dual-light fusion device and the environment-adaptive dual-light fusion method provided in this application achieve the same technical effects, and will not be described further here.

[0131] Continue to refer to Figure 1As shown in the embodiments of this application, the environment-adaptive dual-light fusion device further includes an infrared camera 13 and a visible light camera 14 respectively connected to the processor 11, and further includes a display 16. The infrared camera 13 is used to acquire infrared images and send them to the processor 11. The visible light camera 14 is used to acquire visible light images and send them to the processor 11. The display 16 is used to display the dual-light fusion image; in some embodiments, the display 16 is specifically used to display a dual-light fusion image containing target detection results. The optical axes of the infrared camera 13 and the visible light camera 14 are parallel. Preferably, the infrared camera 13 and the visible light camera 14 are configured as dual-light cameras with the same or approximately the same optical axis.

[0132] Please continue reading. Figure 1 As shown, the environment-adaptive dual-light fusion device further includes an environment sensor 15 connected to the processor 11. The environment sensor 15 is used to collect environmental information characterizing the target scene environment and send it to the processor 11. The environment sensor 15 includes at least one of an ambient light sensor 151 and an ambient humidity sensor 152. The ambient light sensor 151 is used to acquire ambient light information of the target scene environment, and the ambient humidity sensor 152 is used to acquire ambient humidity information of the target scene environment.

[0133] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes in the environment-adaptive dual-light fusion method provided in any embodiment of this application, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An environmentally adaptive dual light fusion method, characterized by, include: Acquire infrared and visible light images of the same target scene; wherein the optical axes of the infrared camera used to acquire the infrared images and the visible light camera used to acquire the visible light images are parallel; The system acquires environmental information characterizing the target scene environment, determines the environmental category of the target scene environment based on the environmental information, and obtains the fusion weight of the infrared image and the visible light image based on the environmental category of the target scene environment; the environmental category includes categories that simultaneously include light conditions and humidity conditions, and categories that only include visual range obstacles. Target detection is performed on the infrared image and the visible light image respectively to obtain infrared target detection results and visible light target detection results; Based on the infrared target detection results and the optical axis information of the infrared camera, the infrared target angle of the target in the infrared image relative to the center of the optical axis of the infrared camera is determined; based on the visible light target detection results and the optical axis information of the visible light camera, the visible light target angle of the target in the visible light image relative to the center of the optical axis of the visible light camera is determined; the infrared target angle and the visible light target angle are matched, and when the infrared target angle and the visible light target angle meet a preset matching condition, the corresponding infrared target detection results and the visible light target detection results are registered to obtain a registration result; the registration result is calibrated based on the texture information of the target. Based on the registration results of the calibration, the infrared target detection results and the visible light target detection results registered are fused according to the fusion weights to obtain a dual-light fused image carrying the target detection results.

2. The dual photo-fusion method of claim 1, wherein, The step of acquiring environmental information characterizing the target scene environment, determining the environment category of the target scene environment based on the environmental information, and obtaining the fusion weight of the infrared image and the visible light image based on the environment category of the target scene environment includes: Acquire the ambient light information and ambient humidity information of the target scene environment respectively; Based on feature extraction and recognition of the infrared image and / or the visible light image, determine whether the weather of the target scene is visibility-impairing weather, and obtain the determination result; Based on the ambient light information, the ambient humidity information, and the judgment result, the environmental category of the target scene environment is determined, and the fusion weight of the infrared image and the visible light image is obtained based on the environmental category of the target scene environment.

3. The dual photo-fusion method of claim 2, wherein, The step of determining the environmental category of the target scene environment based on the ambient light information, the ambient humidity information, and the judgment result includes: If the determination result is that the weather of the target scene is visibility-impairing weather, then the environment category of the target scene environment is determined to be a visibility-impairing environment; If the determination result indicates that the weather of the target scene is non-visual-obstruction weather, the environmental category of the target scene environment is determined based on the ambient light information and the ambient humidity information. The environmental category includes at least one of the following: daytime rainy environment, daytime sunny environment, nighttime rainy environment, and nighttime sunny environment.

4. The dual photo-fusion method of claim 1, wherein, The step of acquiring environmental information characterizing the target scene environment, determining the environment category of the target scene environment based on the environmental information, and obtaining the fusion weight of the infrared image and the visible light image based on the environment category of the target scene environment includes: acquiring an environment category selection instruction as the environmental information; Based on the environment category selected by the environment category selection instruction, the fusion weights of the infrared image and the visible light image are determined; The environmental categories include at least one of the following: daytime rainy environment, daytime nighttime environment, nighttime rainy environment, nighttime clear environment, and visual impairment environment.

5. The dual photo-fusion method according to claim 2 or 4, characterized in that, The step of obtaining the fusion weights of the infrared image and the visible light image based on the environment category of the target scene environment includes: The fusion weights of the infrared image and the visible light image are obtained based on the environment category of the target scene environment and the preset mapping relationship between environment category and fusion weight.

6. The dual photo-fusion method of claim 1, wherein, The calibration of the registration result based on the texture information of the target includes: Based on the registration result, the infrared target texture information and the visible light target texture information corresponding to the infrared target detection result and the visible light target detection result on the registration are obtained respectively; The registration result is calibrated based on the infrared target texture information and the visible light target texture information.

7. The dual photo-fusion method of claim 6, wherein, The step of obtaining the infrared target texture information and visible light target texture information corresponding to the paired infrared target detection results and visible light target detection results according to the registration result includes: Based on the registration results, and according to the infrared target detection results and the visible light target detection results registered, as well as the mapping matrix between the infrared image and the visible light image, an infrared target region and a visible light target region with the same resolution are obtained. The texture information of the infrared target region and the visible light target region are obtained respectively to obtain the infrared target texture information and the visible light target texture information.

8. An environmentally adaptive dual-optical fusion device, comprising: It includes a memory and a processor, and an infrared camera and a visible light camera respectively connected to the processor; wherein the optical axes of the infrared camera used to acquire the infrared image and the visible light camera used to acquire the visible light image are parallel; The memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the dual-light fusion method as described in any one of claims 1 to 7.

9. The dual photofusion device of claim 8, wherein, It also includes displays; The infrared camera is used to capture the infrared image and send the infrared image to the processor; The visible light camera is used to capture the visible light image and send the visible light image to the processor; The display is used to display the dual-light fused image; The infrared camera and the visible light camera have their optical axes parallel.

10. The dual photofusion device of claim 8, wherein, It also includes environmental sensors connected to the processor; The environmental sensor is used to collect environmental information characterizing the target scene environment; The environmental sensor includes at least one of an ambient light sensor and an ambient humidity sensor. The ambient light sensor is used to acquire ambient light information of the target scene environment, and the ambient humidity sensor is used to acquire ambient humidity information of the target scene environment.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dual-light fusion method as described in any one of claims 1 to 7.

Images