Vehicle-mounted dual-light fusion method and system

Abstract

The invention discloses a vehicle-mounted dual-light fusion method and system. Comprising a network construction module, a training set construction module, a network training module and a fusion image generation module. According to the dual-optical fusion network based on the advanced perception task, the fusion image generation module and the target detection module are adopted, and the target detection loss is utilized to guide advanced target information to flow back to the fusion image generation module, so that the performance of the advanced vision task on the fusion image is effectively improved; according to the invention, self-adaptive dual-light fusion is carried out by combining respective advantages of infrared light and visible light, so that the method can adapt to more weather conditions and special user scenes, the target detection performance is improved, and the system safety is improved; the dual-light fusion technology can further optimize the detection rate and the false detection rate of target detection, the performance of a sensing module in various user scenes is improved, and the generated vehicle-mounted scene fusion image is more suitable for observation of a human concerned target.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent assisted driving and image fusion, and in particular relates to a vehicle-mounted dual-light fusion method and system. Background Art

[0002] Object detection is one of the important functions of assisted driving and unmanned driving. Among them, modules such as FCW (front vehicle collision warning) and PCW (pedestrian collision warning) all rely on accurate object detection. However, in real user scenarios, various weather environments will appear. Among them, the risk of accidents at night and in rainy days is more than doubled than under normal circumstances, and phenomena such as fog, haze, sandstorms and strong light seriously reduce the visibility of single visible light vision, which brings great difficulties to object detection.

[0003] Infrared cameras detect thermal radiation from objects and are not affected by light. They have a longer detection distance at night, can see through foggy weather, and have higher resolution. Therefore, combining the advantages of infrared light and visible light for adaptive dual-light fusion can adapt to more weather conditions and special user scenarios, improve target detection performance, and increase system security.

[0004] The fusion of infrared and visible light images aims to synthesize a fused image that not only contains significant targets and rich texture details, but also facilitates the completion of advanced visual tasks. However, existing fusion algorithms only focus on the visual quality and statistical indicators of the fused image, while ignoring the needs of advanced visual tasks (such as subsequent target detection, lane line recognition, etc.); and they need to preset scene categories, separate targets according to different scenes, and then fuse them, which is complex and has poor generalization ability; and the production of data sets requires synthetic data, and it is necessary to obtain the central sub-image of the foreground object after mask processing and fuse the centroid sub-image, which makes data processing very complex. Summary of the invention

[0005] For vehicle usage scenarios, a dual-light fusion method is needed that can not only provide rich information, but also significantly improve subsequent advanced visual tasks, such as target detection and lane line recognition, and make targets more clearly visible at night, in glare, and in bad weather, thereby improving driving safety.

[0006] A vehicle-mounted dual-light fusion method for achieving one of the purposes of the present invention comprises:

[0007] Construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm respectively;

[0008] Annotating target frames of the video data of the vehicle-mounted scene captured by the dual-light camera under different weather conditions to obtain a training set; the targets include vehicles and pedestrians;

[0009] Using the training set to train the fusion image generation network to obtain a trained fusion image generation network;

[0010] Using the training set to train the target detection network to obtain a trained target detection network;

[0011] The infrared image and the visible light image to be fused are input into a trained fusion image generation network to obtain a fusion image; and the fusion image is input into a trained target detection network to obtain a fusion image including a target detection result.

[0012] Furthermore, the fusion image generation network includes:

[0013] The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps whose channel numbers are the first channel values;

[0014] A second feature extraction module; used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a second channel value as the channel number, wherein the second channel value is greater than the first channel value; the high-frequency information includes edges and textures of the image;

[0015] The fused image generation module is used to fuse and reduce the dimensions of the two feature maps whose channel numbers are the second channel values ​​to generate a final fused image.

[0016] Furthermore, in the second feature extraction module, a method for obtaining a feature map having a channel number equal to a second channel value includes:

[0017] The initial feature map whose input channel number is the first channel value is divided into two paths;

[0018] One path obtains a first feature map having a third channel value through a plurality of M×M convolution operations, ReLU functions, and N×N convolution operations, where the third channel value is greater than the first channel value and less than the second channel value;

[0019] Another way is to generate multiple feature maps of different multiples by performing Laplacian convolution with multiple steps.

[0020] Sampling the multiple feature maps of different multiples to obtain multiple feature maps of the same multiple; fusing the multiple feature maps of the same multiple through an N×N convolution operation to obtain a second feature map with a channel number of the third channel value;

[0021] The first feature map and the second feature map are combined and outputted to obtain a feature map with a channel number equal to the second channel value.

[0022] Furthermore, the method for the fused image generation module to generate the final fused image includes:

[0023] The two feature maps with the second channel value are jointly input into multiple M×M convolution modules and Relu modules to reduce the number of channels to the first channel value; then the features are fused through an N×N convolution operation and Tanh activation function to obtain the final fused image.

[0024] Furthermore, the loss function for training the fusion image generation network includes:

[0025] L_content=L_int+α×L_texture

[0026] L_content is the total loss function of the fused image generation network; it is determined by the weighted sum of L_int loss and L_texture; α is used to balance the loss L_int and L_texture;

[0027] L_int=1 / HW||I_f-max(I_ir, I_v1)||_1;

[0028] Among them, H and W are the width and height of the image respectively; ||·||_1 is a norm; max means taking the maximum value; I_ir and I_vl are the pixel value of the infrared image and the gray value of the visible light respectively.

[0029] Furthermore, the loss function for training the target detection network includes:

[0030] L lv5 =α1L_(bbox_loss)+α2L_(cls_loss)+α3L_(obj_loss)

[0031] Where:

[0032] L lv5 is the loss function of the target detection network;

[0033] L_(bbox_loss) is the true target box regression loss value; α1 is the weight of the true target box regression loss L_(bbox_loss);

[0034] L_(cls_loss) is the category loss value; α2 is the weight of the category loss L_(cls_loss);

[0035] L_(obj_loss) is the confidence loss of the predicted target box; α3 is the weight of the confidence loss L_(obj_loss) of the predicted target box.

[0036] A vehicle-mounted dual-light fusion system for achieving the second objective of the present invention comprises:

[0037] Network construction module: used to construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm;

[0038] A training set construction module is used to annotate the target frames of the video data under different weather conditions in the vehicle scene collected by the dual-light camera to obtain a training set; the targets include vehicles and pedestrians;

[0039] Network training module: used to train the fusion image generation network using the training set to obtain a trained fusion image generation network; also used to train the target detection network using the training set to obtain a trained target detection network;

[0040] The fusion image generation module is used to input the infrared image and the visible light image to be fused into the trained fusion image generation network to obtain a fusion image; and input the fusion image into the trained target detection network to obtain a fusion image including the target detection result.

[0041] Furthermore, in the network construction module, the fusion image generation network includes:

[0042] The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps whose channel numbers are the first channel values;

[0043] A second feature extraction module; used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a second channel value as the channel number, wherein the second channel value is greater than the first channel value; the high-frequency information includes edges and textures of the image;

[0044] The fused image generation module is used to fuse and reduce the dimensions of the two feature maps whose channel numbers are the second channel values ​​to generate a final fused image.

[0045] The beneficial effects of the present invention include:

[0046] The present invention designs a dual-light fusion network based on advanced perception tasks, adopts a fusion image generation module and a target detection module, and uses the target detection loss to guide the high-level target information to flow back to the fusion image generation module, thereby effectively improving the performance of high-level visual tasks on the fusion image;

[0047] The present invention combines the respective advantages of infrared light and visible light for adaptive dual-light fusion, which can adapt to more weather conditions and special user scenarios, improve target detection performance, and increase system safety; dual-light fusion technology can further optimize the detection rate and false detection rate of target detection, improve the performance of the perception module in various user scenarios, and the generated vehicle-mounted scene fusion image is also more in line with the observation of targets of human concern. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the network framework;

[0049] Figure 2 is a schematic diagram of the converged network;

[0050] Figure 3 This is a schematic diagram of the LAPDB module;

[0051] Figure 4 This is a diagram of target box regression for calculating DIOU loss Figure 1 ;

[0052] Figure 5 It is a flow chart of the training steps of the fusion network. DETAILED DESCRIPTION

[0053] The following specific implementations are used to explain the technical solutions of the claims of the present invention so that those skilled in the art can understand the claims. The protection scope of the present invention is not limited to the following specific implementation structures. The technical solutions of the claims of the present invention made by those skilled in the art but different from the following specific implementations are also within the protection scope of the present invention.

[0054] Example 1

[0055] A vehicle-mounted dual-light fusion method

[0056] Step 1: Create an infrared and visible light fusion detection dataset

[0057] Using a dual-light camera that registers visible light and infrared images, video data from vehicle-mounted scenes is collected, covering as many weather conditions as possible, such as night, day, sunny, rainy, foggy, snowy, etc., as well as different road scenes, such as urban roads, rural roads, and high-speed scenes, to ensure the generalization ability of the model. By annotating the target boxes of vehicles, pedestrians, and other targets on each frame of the video data collected by the dual-light camera as the training label true value, a training set is finally generated. The training label true value refers to the correct answer or true state of each sample in the data set. These labels are usually provided by human experts and represent the accurate results that should be achieved when the model learns to predict. The true value label provides a "standard answer" for the model's prediction, allowing the model to evaluate the performance of the model by comparing the prediction results with the training label true value, and adjust its own parameters accordingly to optimize the prediction accuracy.

[0058] During the labeling process, target frames are annotated for vehicles, pedestrians and other targets in infrared and visible light respectively. If the target is visible in both infrared and visible light, only one target frame needs to be annotated to reduce redundant information and improve labeling efficiency. If the target is only visible in infrared or only visible in visible light, it is necessary to annotate its target frame separately.

[0059] Step 2: Input the training set into Figure 1 The fusion image generation network based on the deep learning network shown;

[0060] like Figure 2 As shown in the figure, given a pair of matched infrared images and visible light images, firstly, a 3×3 convolution module and a Relu (Rectified Linear Unit) module are used to convert the 3 channels into 16 channels to extract the low-level features of the image; this step converts the 3 color channels of the original image (for visible light images) or the single channel of the infrared image (expanded to 3 channels in some way to match the visible light image) into two 16-channel initial feature maps to increase the representation capability of the data, which is helpful for subsequent processing;

[0061] Then, the Laplacian residual block (LAPRB) is used to extract features and convert the number of channels of the two 16-channel initial feature maps to 48 channels, thereby obtaining two 48-channel feature maps. The LAPRB module uses certain characteristics of the Laplacian operator to extract high-frequency information (such as edges, textures, etc.) in the image, which is very critical for image fusion. Through this module, the number of channels of the input feature map is increased from 16 to 48, further enriching the representation of features; specifically, it includes the following:

[0062] like Figure 3 As shown, Figure 1The 16-channel feature map of the sample size is input into the LAPRB module in two ways. One way passes through two 3×3 convolution modules and a Relu module, and then a 1×1 convolution increases the number of channels to 24 channels to obtain a 24-channel first feature map; the other way passes through Laplacian convolution with 1, 2, and 4 steps to generate feature maps of different scales, and the image size becomes 1, 0.5, and 0.25 times the original size respectively. The 1-fold feature map is merged with the 2-fold (Upsample×2) and 4-fold (Upsample×4) feature maps obtained by upsampling (that is, the smaller feature map is upsampled to match the size of the larger feature map), and a 1×1 convolution is performed for feature fusion to obtain a 24-channel second feature map. Finally, the two 24-channel first feature maps are merged with the second feature map to obtain a 48-channel feature map.

[0063] The two 48-channel feature maps from the infrared image and the visible light image are jointly input into three 3×3 convolution modules and a ReLU module, such as Figure 2 As shown in the figure, the number of channels is gradually reduced from 96 channels to 48 channels, 32 channels and 16 channels. This process realizes the fusion and dimensionality reduction of features, gradually integrates and refines features, and reduces the amount of calculation. Finally, the feature map of 16 channels is fused through a 1×1 convolution and Tanh activation function to obtain the final fused image. Among them, 1×1 convolution is usually used to change the number of channels of data or adjust the data dimension to match the output requirements, and Tanh activation function is used to map the output value to the range of [-1, 1] to generate the final fused image.

[0064] The entire process effectively combines the information of infrared images and visible light images through deep learning methods, especially convolutional neural networks, to generate a fused image that contains both the temperature or thermal radiation information of the infrared image and the details and color information of the visible light image.

[0065] The fused image is input into a target detection network based on the YOLOv5 (You Only Look Once version 5) algorithm for training to obtain a trained target detection network.

[0066] YOLOv5 can accurately locate and identify different target objects in an image. It directly predicts the bounding box and category probability of the target through a convolutional neural network; divides the image into a grid system, and each cell is responsible for detecting the object or object whose physical center point falls inside the cell; and uses anchor boxes to predict the bounding box of the target in the image. These anchor boxes are predefined bounding boxes that are compared with the real bounding box of the target and are dynamically adjusted through optimization algorithms to better match targets of different sizes and shapes.

[0067] Deep learning generally requires the design of a loss function for training. The error between the fitting effect of deep learning and the true value is evaluated through the loss function. The error is reduced by optimizing the function to achieve the purpose of training. The loss function of the fusion image generation network is as follows:

[0068] L_content=L_int+α×L_texture(1)

[0069] L_content is the total loss function of the fused image generation network; it is determined by the weighted sum of L_int loss and L_texture.

[0070] Among them, L_int limits the overall grayscale value of the fused image, and L_texture enables the fused image to contain more fine-grained texture features. Among them: α is used to balance the loss L_int and L_texture.

[0071] L_int=1 / HW||I_f-max(I_ir, I_v1)||_1 (2)

[0072] Among them, H and W are the width and height of the image respectively. The size of the infrared image and the visible light image is the same; ||·||_1 is a norm; max means taking the maximum value; I_ir and I_vl are the pixel value of the infrared image and the gray value of the visible light respectively.

[0073] The loss function of the target detection network is L lv5 ;

[0074] The total loss function is: L_joint = L_content + βL lv5 (3)

[0075] Where β = γ(m-1), m is the number of iterations, and γ is a constant used to balance the fusion task loss L_content and the target detection task loss L lv5 ; β gradually increases with the number of training iterations.

[0076] Where L lv5 The calculation of is as follows:

[0077] L lv5 =α1L_(bbox_loss)+α2L_(cls_loss)+α3L_(obj_loss) (4)

[0078] Where: L lv5 is the total loss function of yolov5 ( Figure 1LV5 LOSS shown); α1 is the weight of the true target box regression loss L_(bbox_loss), which is equal to 0.05 in this implementation; α2 is the weight of the category loss L_(cls_loss), which is equal to 0.3 in this implementation; α3 is the weight of the confidence loss L_(obj_loss) of the predicted target box, which is equal to 0.7 in this embodiment;

[0079] L_(bbox_loss) is the ground-truth target box regression loss, using the DIOU loss function shown in the following formula (5.1);

[0080]

[0081] Where:

[0082] B gt =(x1 gt , y1 gt , x2 gt , y2 gt );(x1 gt , y1 gt ) represents the coordinates of the upper left point of the real target box (that is, the true value of the training label mentioned above); (x2 gt , y2 gt ) represents the coordinates of the lower right point of the training label true value target box;

[0083] B = (x1, y2, x2, y2); (x1, y2) represents the coordinates of the upper left point of the detection target box; (x2, y2) represents the coordinates of the lower right point of the detection target box;

[0084] b is the center point of the detection target box; b gt is the center point of the true target box; ρ() is the Euclidean distance, and c is the diagonal length of the minimum circumscribed rectangle covering the predicted target box and the true target box, such as Figure 4 shown.

[0085] L_(cls_loss) is the category loss, which is calculated using the binary cross entropy loss shown in the following formula (6), where y i is the binary label value of the i-th sample 0 or 1 (ground truth), p(y i =1) is the predicted value of the fusion image generation model for the i-th sample, that is, the probability that the model predicts the label value of the i-th sample is 1.

[0086]

[0087] L_(obj_loss) is the confidence loss of the predicted target box, and the CIOU shown in the following formula (7) is used as the confidence loss of the predicted target box.

[0088]

[0089] d is the center point b of the predicted target box and the center point b of the real target box gt The distance, such as Figure 4 As shown; v is a correction factor, which is used to further adjust the loss function to take into account the shape and direction of the target box. The specific calculation method is:

[0090]

[0091] (wG, hG) is the width and height of the real target box; (wP, hP) is the width and height of the predicted target box;

[0092] Step 3: Training Data

[0093] The training steps are as follows Figure 5 As shown, including:

[0094] Step a): Feed the registered pair of visible light and infrared and the annotated data into the fusion image generation network;

[0095] Step b): Training a network requires M iterations. If the number of iterations is less than M, go to step C. Otherwise, the training is completed.

[0096] Step c): Train the fused image generation network: If the number of cycles is less than p, first update the YOLOV5 weights in the target detection network, then update the weights of the fused image generation network. The loss function is (L_joint), and is updated through the Adam optimizer. Figure 1 The update strategy of all parameters in the network is controlled by β in formula (3). β gradually increases with the number of training iterations. The loss weight of the target detection network increases with the number of training times. The generated fusion image output feature map is the fusion image.

[0097] Step d): Train the target detection network: If the number of cycles is less than q, update the parameters of the target detection yolov5 network (L_lv5) SGD optimizer to obtain the target detection sub-network parameters; otherwise, go to step b;

[0098] The above cycle numbers p and q are empirical values;

[0099] Step 4: Input the infrared and visible light images to be fused into the trained fusion image generation network and target detection network. The output 8-bit feature map is the fusion image with target detection results.

[0100] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0101] Example 2

[0102] A vehicle-mounted dual-light fusion method, comprising:

[0103] Construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm respectively;

[0104] Annotating target frames of the video data of the vehicle-mounted scene captured by the dual-light camera under different weather conditions to obtain a training set; the targets include vehicles and pedestrians;

[0105] Using the training set to train the fusion image generation network to obtain a trained fusion image generation network;

[0106] Using the training set to train the target detection network to obtain a trained target detection network;

[0107] Inputting the infrared image and the visible light image to be fused into a trained fusion image generation network to obtain a fusion image; inputting the fusion image into a trained target detection network to obtain a fusion image including a target detection result;

[0108] In this embodiment, the dual light includes visible light and infrared light; the image captured by the dual light camera includes an image captured based on visible light and an image captured based on infrared light.

[0109] In some embodiments, the fused image generation network includes:

[0110] The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps with the channel number being the first channel value; in this embodiment, the first channel value is 16;

[0111] A second feature extraction module; used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a second channel value as the channel number, wherein the second channel value is greater than the first channel value; the high-frequency information includes the edge and texture of the image; in this embodiment, the second channel value is 48;

[0112] The fused image generation module is used to fuse and reduce the dimensions of the two feature maps whose channel numbers are the second channel values ​​to generate a final fused image.

[0113] In some embodiments, in the second feature extraction module, a method for obtaining a feature map having a channel number of the second channel value includes:

[0114] The initial feature map whose input channel number is the first channel value is divided into two paths;

[0115] One path is passed through a plurality of M×M convolution operations, Relu functions and N×N convolution operations to obtain a first feature map having a channel number of a third channel value, wherein the third channel value is greater than the first channel value and less than the second channel value; in this embodiment, the first channel value, the second channel value and the third channel value are 16, 48 and 24 respectively; M is equal to 3;

[0116] Another path generates multiple feature maps of different multiples after Laplacian convolution with multiple steps. In this embodiment, another path generates feature maps of 1, 0.5, and 0.25 times the original after Laplacian convolution with 1, 2, and 4 steps respectively.

[0117] A sampling operation is performed on the multiple feature maps of different multiples to obtain multiple feature maps of the same multiple; the multiple feature maps of the same multiple are subjected to an N×N convolution operation for feature fusion to obtain a second feature map whose number of channels is a third channel value; the sampling operation in this embodiment is an upsampling operation to obtain three feature maps of 1 multiples; the three feature maps of 1 multiples are subjected to a 1×1 convolution operation for feature fusion to obtain a second feature map whose number of channels is a third channel value;

[0118] The first feature map and the second feature map are combined and outputted to obtain a feature map with a channel number equal to the second channel value.

[0119] In some embodiments, the method in which the fused image generation module generates the final fused image includes:

[0120] The two feature maps with the second channel number are jointly input into multiple M×M convolution modules and Relu modules to reduce the number of channels to the first channel value; then feature fusion is performed through 1 N×N convolution operation and Tanh activation function to obtain the final fused image; in this embodiment, M=3, N=1; after 3 M×M convolution operations, the number of channels is gradually reduced to 48 channels, 32 channels and 16 channels.

[0121] In some embodiments, the loss function when training the fused image generation network includes:

[0122] L_content=L_int+α×L_texture

[0123] L_content is the total loss function of the fused image generation network; it is determined by the weighted sum of L_int loss and L_texture; α is used to balance the loss L_int and L_texture;

[0124] L_int=1 / HW||I_f-max(I_ir, I_vl)||_1;

[0125] Among them, H and W are the width and height of the image respectively; ||·||_1 is a norm; max means taking the maximum value; I_ir and I_vl are the pixel value of the infrared image and the gray value of the visible light respectively.

[0126] In some embodiments, the loss function when training the object detection network includes:

[0127] L lv5 =α1L_(bbox-loss)+α2L_(cls_loss)+α3L_(obj_loss)

[0128] Where:

[0129] L lv5 is the loss function of the target detection network;

[0130] L_(bbox_loss) is the true target box regression loss value; α is the weight of the true target box regression loss L_(bbox-loss);

[0131] L_(cls-loss) is the category loss value; α2 is the weight of the category loss L_(cls-loss);

[0132] L_(obj_loss) is the confidence loss of the predicted target box; α3 is the weight of the confidence loss L_(obj_loss) of the predicted target box.

[0133] Example 3

[0134] A vehicle-mounted dual-light fusion system, comprising:

[0135] Network construction module: used to construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm;

[0136] A training set construction module is used to annotate the target frames of the video data under different weather conditions in the vehicle scene collected by the dual-light camera to obtain a training set; the targets include vehicles and pedestrians;

[0137] Network training module: used to train the fusion image generation network using the training set to obtain a trained fusion image generation network; also used to train the target detection network using the training set to obtain a trained target detection network;

[0138] The fusion image generation module is used to input the infrared image and the visible light image to be fused into the trained fusion image generation network to obtain a fusion image; and input the fusion image into the trained target detection network to obtain a fusion image including the target detection result.

[0139] In some embodiments, in the network construction module, the fusion image generation network includes:

[0140] The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps whose channel numbers are the first channel values;

[0141] A second feature extraction module; used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a second channel value as the channel number, wherein the second channel value is greater than the first channel value; the high-frequency information includes edges and textures of the image;

[0142] A fused image generation module: used for fusing and reducing the dimensions of the two feature maps whose channel numbers are the second channel values, to generate a final fused image;

[0143] In this embodiment, the first channel value, the second channel value and the third channel value are 16, 48 and 24 respectively.

[0144] Example 4

[0145] The embodiment of the present invention also includes a computer program product, including a computer program / instruction, which implements any step of the vehicle-mounted dual-light fusion method when executed by a processor.

[0146] Example 5

[0147] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps of the method described in the present invention are implemented, which will not be repeated here.

[0148] The computer-readable storage medium may be the data transmission device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smartmedia card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc., provided on the computer device.

[0149] Furthermore, the computer-readable storage medium may include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data to be output or has been output.

[0150] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A vehicle-mounted dual-light fusion method, characterized in that: include: Construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm respectively; The target frames of the video data of the vehicle scene captured by the dual-light camera under different weather conditions are annotated to obtain a training set; the targets include vehicles and pedestrians; Using the training set to train the fusion image generation network to obtain a trained fusion image generation network; Using the training set to train the target detection network to obtain a trained target detection network; The infrared image and the visible light image to be fused are input into a trained fusion image generation network to obtain a fused image; and the fused image is input into a trained target detection network to obtain a fused image including a target detection result.

2. The vehicle-mounted dual-light fusion method according to claim 1, characterized in that: The fusion image generation network includes: The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps with the channel number being the first channel value; A second feature extraction module is used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a channel number of the second channel value; the high-frequency information includes edges and textures of the image; The fusion image generation module is used to fuse and reduce the dimensions of the two feature maps whose channel number is the second channel value to generate a final fusion image.

3. The vehicle-mounted dual-light fusion method according to claim 2, characterized in that: In the second feature extraction module, the method for obtaining the feature map having a channel number of the second channel value includes: The initial feature map with the input channel number equal to the first channel value is divided into two paths; One path passes through multiple M×M convolution operations, ReLU functions and N×N convolution operations to obtain the first feature map with the number of channels being the third channel value; Another path generates multiple feature maps of different multiples after performing Laplace convolution with multiple steps; Sampling the multiple feature maps of different multiples to obtain multiple feature maps of the same multiple; fusing the multiple feature maps of the same multiple through an N×N convolution operation to obtain a second feature map with a channel number of the third channel value; The first feature map and the second feature map are combined and outputted to obtain a feature map with a channel number equal to the second channel value.

4. The vehicle-mounted dual-light fusion method according to claim 2, characterized in that: The method for the fusion image generation module to generate the final fusion image includes: The two feature maps with the second channel number are jointly input into multiple M×M convolution modules and ReLU modules to change the channel number to the first channel value; then the features are fused through an N×N convolution operation and Tanh activation function to obtain the final fused image.

5. The vehicle-mounted dual-light fusion method according to claim 1, characterized in that: The loss functions for training the fusion image generation network include: L_content=L_int+α×L_texture L_content is the total loss function of the fusion image generation network; it is determined by the weighted sum of L_int loss and L_texture; α is used to balance the loss L_int and L_texture; L_int=1 / HW‖I_f-max(I_ir,I_vl)‖_1; Among them, H and W are the width and height of the image respectively; ‖.‖_1 is a norm; max means taking the maximum value; I_ir and I_vl are the pixel value of the infrared image and the visible light grayscale value respectively.

6. The vehicle-mounted dual-light fusion method according to claim 1, characterized in that: The loss functions for training the target detection network include: L lv5 =α1L_(bbox_loss)+α2L_(cls_loss)+α3L_(obj_loss) Where: L lv5 is the loss function of the target detection network; L_(bbox_loss) is the true target box regression loss value; α1 is the weight of the true target box regression loss L_(bbox_loss); L_(cls_loss) is the category loss value; α2 is the weight of the category loss L_(cls_loss); L_(obj_loss) is the confidence loss of the predicted target box; α3 is the weight of the confidence loss L_(obj_loss) of the predicted target box.

7. A vehicle-mounted dual-light fusion system, characterized in that: include: Network construction module: used to construct a fusion image generation network based on a deep learning network and a target detection network based on a target detection algorithm; A training set construction module is used to annotate target frames of video data collected by a dual-light camera in vehicle scenes under different weather conditions to obtain a training set; the targets include vehicles and pedestrians; Network training module: used to train the fusion image generation network using the training set to obtain a trained fusion image generation network; It is also used to train the target detection network using the training set to obtain a trained target detection network; The fusion image generation module is used to input the infrared image and visible light image to be fused into the trained fusion image generation network to obtain a fusion image; and input the fusion image into the trained target detection network to obtain a fusion image including the target detection result.

8. The vehicle-mounted dual-light fusion method according to claim 7, characterized in that: In the network construction module, the fusion image generation network includes: The first feature extraction module is used to extract features from the input infrared image and the visible light image respectively, and obtain two initial feature maps with the channel number being the first channel value; A second feature extraction module is used to extract features of high-frequency information from the two initial feature maps respectively, to obtain two feature maps with a channel number of the second channel value; the high-frequency information includes edges and textures of the image; The fusion image generation module is used to fuse and reduce the dimensions of the two feature maps whose channel number is the second channel value to generate a final fusion image.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the vehicle-mounted dual-light fusion method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, any step of the vehicle-mounted dual-light fusion method described in claims 1 to 6 is implemented.

Images