Road target detection method, electronic equipment and storage medium

Abstract

The invention discloses a road target detection method, electronic equipment and a storage medium, and belongs to the technical field of target detection. The road target detection method comprises the steps that a multi-modal image data set is acquired and preprocessed, a double-flow feature extraction backbone network is constructed, and feature extraction is performed on a visible light image and an infrared image; a self-adaptive feature alignment fusion module is designed, dynamic fusion of different modal features is realized through feature cascade and a cross attention mechanism, and weights of different modals are adjusted in a self-adaptive manner; a three-branch dynamic parameter enhancement module is designed, defogging and low-illumination enhancement functions are integrated, and scene quality is optimized through a lightweight parameter estimation network; a two-stage training strategy is adopted, in the first stage, the basic feature extraction capacity is established through pre-training, and in the second stage, the scene degradation problem is focused and optimized. The feature complementarity can be improved, and the road target detection problem under the severe weather condition can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and more specifically, to a road target detection method, electronic equipment and storage medium. Background Art

[0002] In the past decade, thanks to the rapid development of deep learning technology, vision-based object detection has made significant breakthroughs in performance and efficiency. As a basic task of computer vision, object detection plays an indispensable role in practical application scenarios such as autonomous driving and traffic monitoring. In these applications, traditional object detection methods mainly rely on visible light cameras to collect image information. In an environment with sufficient light and clear vision, visible light cameras can provide high-quality images, thereby ensuring the accuracy and reliability of object detection. However, in actual road environments, single visible light sensors face many challenges, especially in severe weather conditions such as rain, fog, and strong light, which often lead to missed detection and false detection of object detection, seriously restricting the reliability and practicality of the system.

[0003] In order to solve the limitations of a single visible light sensor, researchers began to explore multimodal perception solutions, among which infrared imaging technology has attracted much attention due to its sensitivity to the thermal radiation characteristics of objects. Infrared sensors can provide clear target contour information under adverse meteorological conditions, forming a good complementary relationship with visible light sensors. At the same time, infrared sensors also have obvious advantages in cost performance, further enhancing the attractiveness of multimodal perception solutions. In the study of visible light-infrared dual-modal detection, image fusion technology is the key to achieving effective integration of two modal information. At present, the main image fusion technologies can be divided into three categories: pixel-level fusion, decision-level fusion, and feature-level fusion. Although pixel-level fusion is easy to implement, it is easy to introduce fusion artifacts in complex environments, affecting the accuracy of detection results; decision-level fusion integrates the output results of two independent detectors, although it avoids the problem of image alignment, but the computational overhead is large and it is difficult to run efficiently in real-time applications. In contrast, feature-level fusion realizes information integration within the network, which not only reduces the computational cost, but also makes full use of the complementary advantages of the two modalities, so it has received extensive attention and research.

[0004] However, there are still some key problems in the existing feature-level fusion methods that need to be solved urgently. First, the image quality problem is particularly prominent in complex environments. The acquired visible light images often have serious quality degradation, which directly affects the performance of feature extraction and leads to insufficient feature expression. Although the image quality can be improved by introducing an image enhancement module, it is still a major challenge to design a lightweight and efficient enhancement strategy under the detection task framework. Secondly, the feature fusion problem is also a difficult problem that cannot be ignored. Features of different modalities have different numerical distributions and expressions. Simple feature splicing or weighted fusion can easily lead to information redundancy or loss. Especially in the process of feature transfer, the features of a certain modality may over-dominate the fusion results, thereby weakening the advantages of dual-modal complementarity. These problems not only restrict the performance improvement of multimodal target detection technology, but also affect its widespread promotion in practical application scenarios. Summary of the invention

[0005] In order to overcome the technical problems existing in the above prior art, the present invention provides a road target detection method, an electronic device and a storage medium.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A road target detection method comprises the following steps:

[0008] Acquire and preprocess a multimodal image dataset, wherein the multimodal image dataset includes a plurality of pairs of visible light images and infrared images;

[0009] Constructing a dual-stream feature extraction backbone network to extract features from the preprocessed visible light image and the infrared image, respectively, to obtain visible light features and infrared features;

[0010] An adaptive feature alignment fusion module is added to the dual-stream feature extraction backbone network to improve the feature fusion of the visible light feature and the infrared feature and train them to obtain pre-trained weights; a three-branch dynamic parameter enhancement module for optimizing scene quality is introduced on the basis of the pre-trained weights to obtain a road object detection model;

[0011] The image to be detected is input into the road target detection model to obtain the road target detection result.

[0012] Further, the obtaining of the multimodal image dataset is specifically as follows:

[0013] A public multimodal image dataset is obtained, in which people, cars, buses, motorcycles, trucks, and street lights are used as labeled objects, and the multimodal image dataset is divided into a training set and a test set in a ratio of 8:2.

[0014] Furthermore, the preprocessing includes image normalization, size adjustment and data enhancement to improve image quality.

[0015] Furthermore, the construction of the dual-stream feature extraction backbone network specifically includes: constructing HGNetv2 backbone networks for feature extraction of the visible light image and the infrared image respectively;

[0016] Wherein, each of the HGNetv2 backbone networks comprises:

[0017] HGStem module, the HGStem module is used for initial feature extraction;

[0018] A plurality of cascaded HGBlock modules, wherein the plurality of HGBlock modules are used for multi-scale feature extraction;

[0019] DWConv module, the DWConv module is used for depth-wise separable convolution processing.

[0020] Furthermore, the adaptive feature alignment fusion module realizes dynamic fusion of features of different modalities through feature cascading and cross-attention mechanism, and adaptively adjusts the weights of different modalities;

[0021] The three-branch dynamic parameter enhancement module integrates dehazing and low-light enhancement functions, and optimizes scene quality through a lightweight parameter estimation network.

[0022] Further, the step of dynamically fusing the visible light feature and the infrared feature by the adaptive feature alignment and fusion module includes:

[0023] For the input visible light feature F vis and infrared signature F inf Calculate the channel mean and channel maximum statistical features respectively, and concatenate the mean feature and the maximum feature in the channel dimension to obtain the initial statistical features. The formula is as follows:

[0024]

[0025] F stat =Concat(F avg ,F max )

[0026] Among them, CAA(·) represents the channel dimension average aggregation operation, which is an operation to calculate the average value of the feature map in the channel dimension; CMA(·) represents the channel dimension maximum aggregation operation, which is an operation to calculate the maximum value of the feature map in the channel dimension; Concat(·) represents channel-by-channel concatenation;

[0027] The statistical features are rearranged into a sequence form, and Q, K and V are calculated respectively through three transformation matrices. vis K with infrared characteristics inf The relevance score is calculated and normalized by Softmax to obtain the attention weight, which is multiplied by V to complete feature enhancement. At the same time, reverse attention calculation is performed to achieve bidirectional feature interaction. The formula is as follows:

[0028]

[0029] in, represents the scaling factor used to adjust the attention score;

[0030] Use a 7×7 convolution kernel to compress the multi-channel features into a single channel and output the weight map through the Sigmoid function:

[0031] F att =σ(Conv(Concat(F att_vis ,F att_inf )))

[0032] Among them, Conv(·) is a 7×7 convolution kernel used to output a single-channel attention map to achieve mapping compression from multi-channel to single-channel; σ(·) is the Sigmoid function used to output the final attention weight; F att is the output weight map;

[0033] The weight map is used to weight the visible light features and infrared features to achieve dynamic balance and fusion of features:

[0034]

[0035] in, Represents a multiplication operation; Conv(·) is a 1×1 point-by-point convolution operation, which is used to compress the number of channels of the input feature map to the number of channels of the original image;

[0036] The residual connection is used to ensure information flow, and the fused feature map is output for subsequent road target detection.

[0037] Further, the step of optimizing the scene quality by the three-branch dynamic parameter enhancement module includes:

[0038] The initial feature extraction of the input visible light image is performed through the shared feature extraction network. The formula is as follows:

[0039]

[0040] Among them, there is a LeakyReLU activation function after each layer of convolution, which enables the network to simultaneously retain the spatial structure information of the image and effectively extract multi-level features from low-level texture to high-level semantics;

[0041] In the dehazing branch, the 5th layer of convolutional network is used to extract features and obtain the global description of the image through global average pooling. The atmospheric light value is obtained through a two-layer fully connected network, and the transmission map is output at the same time. The atmospheric scattering model formula is input to perform image dehazing. The formula is as follows:

[0042]

[0043]

[0044] Among them, t(x) represents the transmission map, A represents the atmospheric light value; fc1 and fc2 represent two fully connected layers; J(x) represents the clear image to be restored; I(x) represents the foggy input image;

[0045] In the low-light enhancement branch, 8 curve parameters are output through the fifth-layer convolutional network structure and sigmoid mapping is performed. The iterative image enhancement strategy is used to perform 8 progressive enhancements. The formula is as follows:

[0046]

[0047] in, i∈{1,2,...,8} means splitting the feature map according to the channel dimension, I t+1 is the enhanced image, I t is the original image, r is the curve parameter map; the constant ε is used to ensure the stability of numerical calculations, and the clamp operation is used to ensure that the pixel values ​​are always kept within a reasonable range during the enhancement process.

[0048] Furthermore, the road object detection model adopts a two-stage training strategy to complete model training, specifically:

[0049] In the first stage of training, the initial learning rate is set to 0.0001, the weight decay rate is set to 0.001, and the batch size is set to 8. When the adaptive feature alignment fusion module is added but the three-branch dynamic parameter enhancement module is not added, the feature extraction network is pre-trained to obtain the initialization parameters and lay the foundation for feature understanding capabilities.

[0050] In the second stage of training, the pre-trained weights of the first stage are loaded, and a three-branch dynamic parameter enhancement module is integrated to optimize the scene quality, while keeping the input image size at 640×640 pixels.

[0051] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned road target detection method when executing the computer program.

[0052] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the road target detection method described above is implemented.

[0053] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0054] The present invention provides a road target detection method, an electronic device and a storage medium. The road target detection method includes: obtaining a multimodal image data set and preprocessing it, constructing a dual-stream feature extraction backbone network, and extracting features from visible light images and infrared images respectively; designing an adaptive feature alignment fusion module, realizing dynamic fusion of different modal features through feature cascade and cross-attention mechanism, and adaptively adjusting the weights of different modalities; designing a three-branch dynamic parameter enhancement module, integrating defogging and low-light enhancement functions, and optimizing scene quality through a lightweight parameter estimation network; adopting a two-stage training strategy, in which the first stage establishes basic feature extraction capabilities through pre-training, and the second stage focuses on optimizing scene degradation problems. The present invention can improve feature complementarity and effectively solve the problem of road target detection under severe weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a flow chart of a road target detection method provided in one embodiment of the present application;

[0057] Figure 2 A schematic diagram of the algorithm structure in the background technology of this application;

[0058] Figure 3 Schematic diagram of the structure of the AFAF module in the embodiment of the present application;

[0059] Figure 4 Schematic diagram of the structure of the TDPE module in the embodiment of the present application;

[0060] Figure 5 This is a diagram of the scene optimization structure in the embodiment of the present application. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0062] Embodiment 1:

[0063] like Figure 1 As shown, the present invention provides a technical solution:

[0064] A road target detection method comprises the following steps:

[0065] Acquire and preprocess a multimodal image dataset, wherein the multimodal image dataset includes a plurality of pairs of visible light images and infrared images;

[0066] Constructing a dual-stream feature extraction backbone network to extract features from the preprocessed visible light image and the infrared image, respectively, to obtain visible light features and infrared features;

[0067] An adaptive feature alignment fusion module (AFAF module) is added to the dual-stream feature extraction backbone network to improve the feature fusion of the visible light feature and the infrared feature and train them to obtain pre-trained weights; a three-branch dynamic parameter enhancement module (TDPE module) for optimizing scene quality is introduced on the basis of the pre-trained weights to obtain a road object detection model;

[0068] The image to be detected is input into the road target detection model to obtain the road target detection result.

[0069] Embodiment 2:

[0070] Based on Example 1, the step of acquiring the multimodal image dataset is as follows:

[0071] A public multimodal image dataset is obtained, in which people, cars, buses, motorcycles, trucks, and street lights are used as labeled objects, and the multimodal image dataset is divided into a training set and a test set in a ratio of 8:2.

[0072] Furthermore, the multimodal image datasets are the public multimodal multi-scene object detection datasets M3FD and MSRS;

[0073] The M3FD dataset contains 4177 pairs of infrared and visible light images, covering different weather conditions such as day, night, rain and fog, as well as traffic scenarios in different seasons;

[0074] The MSRS dataset contains 1,444 pairs of strictly aligned infrared and visible light images. The dataset covers road scenes during the day and at night.

[0075] In order to expand the training samples, combined with the diversity of the M3FD dataset in different weather conditions and scenes and the high-quality image features of the optimized MSRS dataset, the two datasets were combined to form a training set of 3415 pairs of images and a test set of 854 pairs of images.

[0076] Furthermore, the preprocessing includes image normalization, size adjustment and data enhancement to improve image quality.

[0077] Furthermore, the construction of the dual-stream feature extraction backbone network specifically includes: constructing HGNetv2 backbone networks for feature extraction of the visible light image and the infrared image respectively;

[0078] like Figure 2 As shown, each of the HGNetv2 backbone networks includes:

[0079] HGStem module, the HGStem module is used for initial feature extraction;

[0080] A plurality of cascaded HGBlock modules, wherein the plurality of HGBlock modules are used for multi-scale feature extraction;

[0081] DWConv module, the DWConv module is used for depth-wise separable convolution processing.

[0082] Further, the HGStem module consists of a 3×3 convolutional layer with a stride of 2, a BatchNorm layer, a ReLU activation function, and a maximum pooling layer with a stride of 2;

[0083] The HGBlock module includes a main branch and a skip connection branch. The main branch is a 3×3 depthwise separable convolution, a BatchNorm layer, a ReLU activation function, and a 1×1 convolution for channel adjustment. The skip connection branch uses a 1×1 convolution for adjustment when the number of input and output channels is different, and a convolution with a step size of 2 is used when the feature map size needs to be changed.

[0084] The C1 feature map of visible light and infrared images is output by the HGStem module respectively, and then the C2, C3, C4, and C5 feature maps are obtained in sequence through four HGBlock modules. The number of feature map channels is 64, 128, 256, and 512 respectively. After each HGBlock module, the SE attention mechanism is added for channel weighting and the residual connection is used to maintain gradient propagation. Finally, the feature pyramid structure is used to upsample the high-level features and fuse them with the low-level features to generate a multi-scale feature representation.

[0085] Furthermore, the adaptive feature alignment fusion module realizes dynamic fusion of features of different modalities through feature cascading and cross-attention mechanism, and adaptively adjusts the weights of different modalities;

[0086] The three-branch dynamic parameter enhancement module integrates dehazing and low-light enhancement functions, and optimizes scene quality through a lightweight parameter estimation network.

[0087] Further, the step of dynamically fusing the visible light feature and the infrared feature by the adaptive feature alignment and fusion module includes:

[0088] like Figure 3 As shown, for the input visible light feature F vis and infrared signature F inf Calculate the channel mean and channel maximum statistical features respectively, and concatenate the mean feature and the maximum feature in the channel dimension to obtain the initial statistical features. The formula is as follows:

[0089]

[0090] F stat =Concat(F avg ,F max )

[0091] Among them, CAA(·) represents the channel dimension average aggregation operation, which is an operation to calculate the average value of the feature map in the channel dimension; CMA(·) represents the channel dimension maximum aggregation operation, which is an operation to calculate the maximum value of the feature map in the channel dimension; Concat(·) represents channel-by-channel concatenation, which allows the network to adaptively learn how to weigh and fuse these features in subsequent layers;

[0092] The statistical features are rearranged into a sequence form, and Q, K and V are calculated respectively through three transformation matrices. vis K with infrared characteristics inf The relevance score is calculated and normalized by Softmax to obtain the attention weight, which is multiplied by V to complete feature enhancement. At the same time, reverse attention calculation is performed to achieve bidirectional feature interaction. The formula is as follows:

[0093]

[0094] in, represents the scaling factor used to adjust the attention score;

[0095] Use a 7×7 convolution kernel to compress the multi-channel features into a single channel and output the weight map through the Sigmoid function:

[0096] F att=σ(Conv(Concat(F att_vis ,F att_inf )))

[0097] Among them, Conv(·) is a 7×7 convolution kernel used to output a single-channel attention map to achieve mapping compression from multi-channel to single-channel; σ(·) is the Sigmoid function used to output the final attention weight; F att is the output weight map;

[0098] The weight map is used to weight the visible light features and infrared features to achieve dynamic balance and fusion of features:

[0099]

[0100] in, Represents a multiplication operation; Conv(·) is a 1×1 point-by-point convolution operation, which is used to compress the number of channels of the input feature map to the number of channels of the original image;

[0101] The residual connection is used to ensure information flow, and the fused feature map is output for subsequent road target detection.

[0102] Further, the step of optimizing the scene quality by the three-branch dynamic parameter enhancement module includes:

[0103] like Figure 4 As shown, the initial feature extraction of the input visible light image is performed through the shared feature extraction network, and the formula is as follows:

[0104]

[0105] Among them, there is a LeakyReLU activation function after each layer of convolution, which enables the network to simultaneously retain the spatial structure information of the image and effectively extract multi-level features from low-level texture to high-level semantics;

[0106] In the dehazing branch, the 5th layer of convolutional network is used to extract features and obtain the global description of the image through global average pooling. The atmospheric light value is obtained through a two-layer fully connected network, and the transmission map is output at the same time. The atmospheric scattering model formula is input to perform image dehazing. The formula is as follows:

[0107]

[0108]

[0109]

[0110] Among them, t(x) represents the transmission map, A represents the atmospheric light value; fc1 and fc2 represent two fully connected layers; J(x) represents the clear image to be restored; I(x) represents the foggy input image;

[0111] In the low-light enhancement branch, 8 curve parameters are output through the fifth-layer convolutional network structure and sigmoid mapping is performed. The iterative image enhancement strategy is used to perform 8 progressive enhancements. The formula is as follows:

[0112]

[0113]

[0114]

[0115] in, i∈{1,2,…,8} means splitting the feature map according to the channel dimension, I t+1 is the enhanced image, I t is the original image, r is the curve parameter map; the constant ε is used to ensure the stability of numerical calculations, and the clamp operation is used to ensure that the pixel values ​​are always kept within a reasonable range during the enhancement process;

[0116] Finally, perform scene optimization; Figure 5 As shown, the above defogging process is performed first to avoid low-light enhancement exposure affecting the defogging effect; then low-light enhancement is performed to obtain optimized image feature input and sent to the subsequent target detection network;

[0117] Among them, the feature extraction network serves the transmission map and atmospheric light value estimation at the same time in the early stage, realizing the sharing of network parameters and effectively reducing the computational overhead.

[0118] Furthermore, the road object detection model adopts a two-stage training strategy to complete model training, specifically:

[0119] In the first stage of training, the initial learning rate is 0.0001, the weight decay rate is 0.001, and the batch size is 8. When the AFAF module is added but the TDPE module is not added, the feature extraction network is pre-trained to obtain reasonable parameter initialization and establish basic feature understanding capabilities;

[0120] Then in the second stage of training, the optimization focus is shifted to scene optimization. The pre-trained weights of the first stage are loaded and the TDPE module is introduced for scene quality optimization, so that the network can better handle scene degradation problems. During this period, the input image size is kept at 640×640 pixels.

[0121] Furthermore, in this application, all experiments were conducted under the Ubuntu 20.04 operating system, the GPU was NVIDIA GeForce GTX 3090, the video memory was 24GB, the CPU was Intel (R) Xeon (R) Gold 5418Y, the deep learning architecture was Pytorch 2.1.0, and the programming language was Python 3.10;

[0122] During training, the initial learning rate was set to 0.0001, the AdamW optimizer with a weight decay of 0.0001 was used, the batch_size was set to 8, and the number of training rounds was set to 300;

[0123] After training is completed, the optimal weights are used to test the test set to evaluate the performance of the model;

[0124] After obtaining the experimental results, comparison and analysis were performed. The experimental part used mAP0.5, mAP0.5:0.95 and model size as evaluation indicators. In order to verify the superiority of the improved strategy proposed in this embodiment over the current mainstream two-stream target detection algorithm, a comparative experiment was conducted. The experimental results are shown in Table 1 below.

[0125] Table 1. Comparative experimental results:

[0126]

[0127] Embodiment 3:

[0128] The present invention provides a technical solution:

[0129] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned road target detection method when executing the computer program.

[0130] In this embodiment, the computer system suitable for implementing the electronic device of the embodiment of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM), such as executing the method described in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other through a bus. Input / output (I / O) interface is also connected to the bus.

[0131] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage part as needed.

[0132] Embodiment 4:

[0133] The present invention provides a technical solution:

[0134] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer-readable storage medium implements the aforementioned road object detection method. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0139] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the application point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0140] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0141] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be construed as a limitation to the claims. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A road target detection method, characterized in that: The following steps are involved: Acquire and preprocess a multimodal image dataset, wherein the multimodal image dataset includes a plurality of pairs of visible light images and infrared images; Constructing a dual-stream feature extraction backbone network to extract features from the preprocessed visible light image and the infrared image, respectively, to obtain visible light features and infrared features; An adaptive feature alignment fusion module is added to the dual-stream feature extraction backbone network to improve the feature fusion of the visible light feature and the infrared feature and train them to obtain pre-trained weights; a three-branch dynamic parameter enhancement module for optimizing scene quality is introduced on the basis of the pre-trained weights to obtain a road object detection model; The image to be detected is input into the road target detection model to obtain the road target detection result.

2. The road target detection method according to claim 1, characterized in that: The step of acquiring the multimodal image dataset is as follows: A public multimodal image dataset is obtained, in which people, cars, buses, motorcycles, trucks, and street lights are used as labeled objects, and the multimodal image dataset is divided into a training set and a test set in a ratio of 8:

2.

3. The road target detection method according to claim 1, characterized in that: The preprocessing includes image normalization, size adjustment and data enhancement to improve image quality.

4. The road target detection method according to claim 1, characterized in that: The construction of the dual-stream feature extraction backbone network specifically includes: constructing HGNetv2 backbone networks for feature extraction of the visible light image and the infrared image respectively; Wherein, each of the HGNetv2 backbone networks comprises: HGStem module, the HGStem module is used for initial feature extraction; A plurality of cascaded HGBlock modules, wherein the plurality of HGBlock modules are used for multi-scale feature extraction; DWConv module, the DWConv module is used for depth-wise separable convolution processing.

5. The road target detection method according to claim 1, characterized in that: The adaptive feature alignment fusion module realizes the dynamic fusion of different modal features through feature cascading and cross attention mechanism, and adaptively adjusts the weights of different modalities; The three-branch dynamic parameter enhancement module integrates dehazing and low-light enhancement functions, and optimizes scene quality through a lightweight parameter estimation network.

6. The road target detection method according to claim 5, characterized in that: The step of dynamically fusing the visible light feature and the infrared feature by the adaptive feature alignment and fusion module comprises: For the input visible light feature F vis and infrared signature F inf Calculate the channel mean and channel maximum statistical features respectively, and concatenate the mean feature and the maximum feature in the channel dimension to obtain the initial statistical features. The formula is as follows: F stat =Concat(F avg ,F max ) Among them, CAA(·) represents the channel dimension average aggregation operation, which is an operation to calculate the average value of the feature map in the channel dimension; CMA(·) represents the channel dimension maximum aggregation operation, which is an operation to calculate the maximum value of the feature map in the channel dimension; Concat(·) represents channel-by-channel concatenation; The statistical features are rearranged into a sequence form, and Q, K and V are calculated respectively through three transformation matrices. vis K with infrared characteristics inf The relevance score is calculated and normalized by Softmax to obtain the attention weight, which is multiplied by V to complete feature enhancement. At the same time, reverse attention calculation is performed to achieve bidirectional feature interaction. The formula is as follows: in, represents the scaling factor used to adjust the attention score; Use a 7×7 convolution kernel to compress the multi-channel features into a single channel and output the weight map through the Sigmoid function: F att =σ(Conv(Concat(F att_vis ,F att_inf ))) Among them, Conv(·) is a 7×7 convolution kernel used to output a single-channel attention map to achieve mapping compression from multi-channel to single-channel; σ(·) is the Sigmoid function used to output the final attention weight; F att is the output weight map; The weight map is used to weight the visible light features and infrared features to achieve dynamic balance and fusion of features: in, Represents a multiplication operation; Conv(·) is a 1×1 point-by-point convolution operation, which is used to compress the number of channels of the input feature map to the number of channels of the original image; The residual connection is used to ensure information flow, and the fused feature map is output for subsequent road target detection.

7. The road target detection method according to claim 5, characterized in that: The steps of optimizing the scene quality by the three-branch dynamic parameter enhancement module include: The initial feature extraction of the input visible light image is performed through the shared feature extraction network. The formula is as follows: Among them, there is a LeakyReLU activation function after each layer of convolution, which enables the network to simultaneously retain the spatial structure information of the image and effectively extract multi-level features from low-level texture to high-level semantics; In the dehazing branch, the 5th layer of convolutional network is used to extract features and obtain the global description of the image through global average pooling. The atmospheric light value is obtained through a two-layer fully connected network, and the transmission map is output at the same time. The atmospheric scattering model formula is input to perform image dehazing. The formula is as follows: Among them, t(x) represents the transmission map, A represents the atmospheric light value; fc1 and fc2 represent two fully connected layers; J(x) represents the clear image to be restored; I(x) represents the foggy input image; In the low-light enhancement branch, 8 curve parameters are output through the fifth-layer convolutional network structure and sigmoid mapping is performed. The iterative image enhancement strategy is used to perform 8 progressive enhancements. The formula is as follows: in, Indicates that the feature map is split according to the channel dimension, I t+1 is the enhanced image, I t is the original image, r is the curve parameter map; the constant ε is used to ensure the stability of numerical calculations, and the clamp operation is used to ensure that the pixel values ​​are always kept within a reasonable range during the enhancement process.

8. The road target detection method according to claim 1, characterized in that: The road object detection model adopts a two-stage training strategy to complete model training, specifically: In the first stage of training, the initial learning rate is set to 0.0001, the weight decay rate is set to 0.001, and the batch size is set to 8. When the adaptive feature alignment fusion module is added but the three-branch dynamic parameter enhancement module is not added, the feature extraction network is pre-trained to obtain the initialization parameters and lay the foundation for feature understanding capabilities. In the second stage of training, the pre-trained weights of the first stage are loaded, and a three-branch dynamic parameter enhancement module is integrated to optimize the scene quality, while keeping the input image size at 640×640 pixels.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the road object detection method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the road object detection method according to any one of claims 1 to 8 is implemented.

Images