Intelligent driving lightweight target detection method and device

Abstract

The application provides a kind of intelligent driving light weight target detection method and device, it is related to intelligent driving environment perception technical field.The method comprises: obtaining intelligent driving scene image;It is input light weight single-stage target detection (You Only Look Once, YOLO) network, the network includes backbone network, neck network and detection head, neck network includes C2f-Fb light weight feature fusion module, ADown light weight down-sampling module and DySample dynamic up-sampling module;Obtain multiple scale initial feature maps by backbone network extraction;Multi-scale feature fusion is carried out by neck network, and multiple scale fusion feature maps are obtained;The traffic target detection result is output by detection head.C2f-Fb module reduces the feature fusion calculation amount by FasterBlock unit, ADown module generates down-sampling feature map, and DySample module generates high-resolution feature map.The application can reduce network parameter quantity and calculation amount, improve the applicability of real-time target detection of vehicle-mounted terminal.

Description

Technical Field

[0001] This application relates to the field of intelligent driving environment perception technology, and in particular to a traffic target detection technology based on a lightweight target detection network. Background Technology

[0002] Intelligent driving vehicles typically rely on functional modules such as environmental perception, path planning, and vehicle control to achieve autonomous driving. Among these, the environmental perception module acquires information about the road environment surrounding the vehicle, providing a data foundation for subsequent path planning and vehicle control. Object detection is a crucial technology in environmental perception, primarily used to identify traffic targets such as pedestrians, motor vehicles, and non-motorized vehicles around the vehicle and determine their positions in the image. The accuracy and real-time performance of object detection results directly impact the intelligent driving vehicle's ability to understand its surroundings and the safety of subsequent driving decisions.

[0003] With the development of deep learning and machine vision technologies, the You Only LookOnce (YOLO) single-stage object detection algorithm has been widely used in traffic scene object detection tasks due to its fast detection speed and high detection accuracy. Existing YOLOv8 and its improved models can meet the needs of road object recognition and localization to a certain extent, but when deployed on vehicle-mounted devices, they still suffer from problems such as a large number of model parameters, high floating-point computation requirements, and a strong dependence on hardware computing power and storage resources during the inference process. Because vehicle-mounted embedded computing devices are typically limited by power consumption, size, heat dissipation, and computing resources, complex object detection models are prone to high inference latency and insufficient real-time performance in actual operation, making it difficult to stably meet the real-time obstacle detection requirements of intelligent driving scenarios.

[0004] Furthermore, existing object detection networks typically require downsampling and upsampling during multi-scale feature fusion. Traditional convolutional downsampling structures, while compressing feature map size and expanding the receptive field, often introduce more convolutional computations, increasing the overall computational burden of the network. Conventional static upsampling methods usually employ fixed interpolation rules, making it difficult to adaptively adjust the sampling position based on the texture distribution, target edges, and target scale variations of the input image. In complex traffic scenes such as urban roads, targets such as pedestrians, non-motorized vehicles, and small vehicles exhibit large scale variations and are prone to losing edge details. Static upsampling methods can easily lead to insufficient recovery of feature details, thus affecting the detection performance of multi-scale traffic targets.

[0005] Therefore, how to reduce the number of parameters and computational load of the target detection network while ensuring the accuracy of traffic target detection, and improve its real-time inference capability on in-vehicle embedded computing devices, is an urgent problem to be solved in intelligent driving target detection technology. Summary of the Invention

[0006] To address the issues of large parameter count, high computational cost, insufficient real-time inference, and poor multi-scale traffic target feature reconstruction in existing YOLO object detection networks deployed on intelligent driving vehicles, this application provides a lightweight object detection method and device for intelligent driving. It introduces a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module into the neck network to maintain traffic target detection capabilities while reducing network computational overhead.

[0007] Firstly, this application provides a lightweight target detection method for intelligent driving, comprising: Acquire images of intelligent driving scenarios; The intelligent driving scene image is input into a lightweight YOLO target detection network, which includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The backbone network is used to extract features from the intelligent driving scene images to obtain initial feature maps at multiple scales. The neck network is used to perform multi-scale feature fusion on the initial feature maps of the multiple scales to obtain a fused feature map of the multiple scales. The C2f-Fb lightweight feature fusion module is used to split the input feature map into features and extract lightweight features from at least one branch feature through the FasterBlock unit, and then fuse it with the features of the remaining branches. The FasterBlock unit includes partial channel convolution and 1×1 convolution. The partial channel convolution is used to extract spatial features from a portion of the input feature map and allows the remaining channels that do not participate in the spatial feature extraction to skip the partial channel convolution as directly connected channels. The 1×1 convolution is used to receive the channel features processed by the partial channel convolution and the channel features of the directly connected channels, and perform channel dimension feature fusion. During the downsampling process of the neck network, the ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map, and to concatenate the dual-branch output features according to the channel dimension to obtain the downsampled feature map used for the multi-scale feature fusion. During the upsampling process of the neck network, the DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for the multi-scale feature fusion. The detection head detects the fused feature maps of multiple scales output by the neck network and outputs traffic target detection results, which include the category and location of the traffic targets.

[0008] Optionally, the C2f-Fb lightweight feature fusion module includes a feature splitting unit, at least one FasterBlock unit, and a feature fusion unit connected in sequence. The feature splitting unit is used to divide the input feature map into retained branch features and processed branch features. After the processed branch features are extracted by the at least one FasterBlock unit using lightweight features, they are fused with the retained branch features by the feature fusion unit to obtain a fused output feature map.

[0009] Optionally, the FasterBlock unit includes a 3×3 partial channel convolution layer and two 1×1 ordinary convolution layers arranged sequentially; wherein, the 3×3 partial channel convolution is used to perform spatial convolution only on a portion of the continuous channels of the input feature map, and allows the remaining channels that do not participate in the spatial convolution to skip the 3×3 partial channel convolution as directly connected channels; the two 1×1 ordinary convolution layers are used to receive the channel features processed by the 3×3 partial channel convolution and the channel features of the directly connected channels, and perform channel dimension fusion on the two.

[0010] Optionally, the ADown lightweight downsampling module performs 2×2 average pooling with a stride of 1 and no padding on the input feature map to obtain a pooled feature map; the pooled feature map is divided into a first branch feature and a second branch feature; the first branch feature is processed by 3×3 convolution, and the number of channels is adjusted to half of the total number of target channels; the second branch feature is processed by 1×1 convolution and max pooling downsampling, wherein the max pooling downsampling uses 3×3 max pooling with a stride of 2 and padding of 1; the processed first branch feature and second branch feature are concatenated according to the channel dimension to obtain the downsampled feature map.

[0011] Optionally, the DySample dynamic upsampling module generates a sampling offset based on the input feature map, comprising: inputting the input feature map into a sampling point generator, the sampling point generator including a range factor branch and an offset prediction branch; performing a linear mapping on the input feature map through the range factor branch and normalizing it through a sigmoid activation function to obtain a sampling range factor; performing a linear mapping on the input feature map through the offset prediction branch to obtain an offset prediction amount; and obtaining a constrained sampling offset based on the sampling range factor and the offset prediction amount.

[0012] Optionally, obtaining the constrained sampling offset based on the sampling range factor and the offset prediction includes: multiplying the sampling range factor, the offset prediction, and a preset scaling factor to obtain the constrained sampling offset; wherein the preset scaling factor is 0.5.

[0013] Optionally, the DySample dynamic upsampling module adds the constrained sampling offset to the basic sampling grid to obtain the dynamic sampling position, and resamples the input feature map through a bilinear interpolation resampling operation based on the sampling grid to generate the high-resolution feature map.

[0014] Optionally, the step of detecting the fused feature maps of multiple scales output by the neck network through the detection head and outputting traffic target detection results includes: performing target classification prediction and target location regression on the fused feature maps of different scales through the detection head to obtain the category, location and confidence of the traffic target; wherein the traffic target includes at least one of vehicles, pedestrians and non-motorized vehicles.

[0015] Optionally, the trained lightweight YOLO object detection network is deployed to an in-vehicle embedded computing device, and the in-vehicle embedded computing device performs object detection on real-time acquired intelligent driving scene images, and outputs the traffic object detection results.

[0016] Secondly, this application provides a lightweight target detection device for intelligent driving, comprising: The image acquisition module is used to acquire images of intelligent driving scenarios; The network input module is used to input the intelligent driving scene image into the lightweight YOLO target detection network. The lightweight YOLO target detection network includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The feature extraction module is used to extract features from the intelligent driving scene image through the backbone network to obtain initial feature maps at multiple scales. A multi-scale feature fusion module is used to perform multi-scale feature fusion on the initial feature maps of multiple scales through the neck network to obtain a fused feature map of multiple scales. Specifically, the C2f-Fb lightweight feature fusion module is used to perform feature splitting on the input feature map, and after extracting lightweight features from at least one branch feature through a FasterBlock unit, it is fused with the remaining branch features. The ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map, and concatenate the dual-branch output features according to the channel dimension to obtain a downsampled feature map for the multi-scale feature fusion. The DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for the multi-scale feature fusion. The detection output module is used to detect the multi-scale fused feature map output by the neck network through the detection head, and output traffic target detection results, which include the category and location of the traffic target.

[0017] The beneficial effects of this application include at least the following: This application incorporates a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module into the neck network of a lightweight YOLO object detection network, enabling all three stages—feature fusion, downsampling, and upsampling—to participate in lightweight transformation. This reduces redundant convolutional computations, lowers the number of network parameters and floating-point operations, and improves the network's real-time inference capabilities on in-vehicle embedded computing devices during multi-scale traffic object detection.

[0018] This application uses a lightweight C2f-Fb feature fusion module to perform feature splitting on the input feature map, and extracts spatial features only from a subset of channels through partial channel convolution in the FasterBlock unit. The remaining channels, not involved in spatial feature extraction, are treated as directly connected channels and skip the partial channel convolution. Then, the channel features processed by the partial channel convolution are combined with the channel features of the directly connected channels into a 1×1 convolution for channel-dimensional feature fusion. This reduces the computational cost of spatial convolution while preserving effective feature transfer capabilities, thus lowering the computational overhead of the neck network feature fusion stage.

[0019] This application utilizes the ADown lightweight downsampling module, employing pooling and dual-branch convolution to generate downsampled feature maps. This approach preserves feature information from different branches while compressing the feature map size, reducing the computational burden of conventional convolutional downsampling. Furthermore, the DySample dynamic upsampling module generates sampling offsets based on the input feature map and determines the dynamic sampling position using the base sampling grid. This allows the upsampling process to adaptively adjust the sampling position according to image features, improving the ability of high-resolution feature maps to reproduce the edges and details of traffic targets. Consequently, this approach helps maintain the detection performance of multi-scale traffic targets such as vehicles, pedestrians, and non-motorized vehicles. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the lightweight YOLO target detection network structure provided in the embodiments of this application; Figure 2 A flowchart of the intelligent driving lightweight target detection method provided in the embodiments of this application; Figure 3 A schematic diagram of the C2f-Fb lightweight feature fusion module structure provided in this application embodiment; Figure 4 This is a schematic diagram of the FasterBlock cell structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of a partial channel convolution structure provided in an embodiment of this application; Figure 6 This is a schematic diagram of the ADown lightweight downsampling module structure provided in an embodiment of this application; Figure 7 This is a schematic diagram of the DySample dynamic upsampling module structure provided in an embodiment of this application; Figure 8 This is a structural block diagram of the intelligent driving lightweight target detection device provided in an embodiment of this application. Detailed Implementation

[0021] The present application will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the scope of protection of the present application.

[0022] This embodiment provides a lightweight target detection method for intelligent driving, which is implemented based on a lightweight single-stage target detection (You Only Look Once, YOLO) network. For ease of description, the lightweight single-stage target detection network will be referred to as the lightweight YOLO target detection network below. This method can be applied to intelligent driving vehicles, driver-assisted vehicles, in-vehicle environmental perception systems, or in-vehicle embedded computing devices to detect traffic targets in the vehicle's driving environment. The traffic targets may include at least one of vehicles, pedestrians, and non-motorized vehicles. The vehicles may include road traffic participation targets such as motor vehicles and motorcycles, and the non-motorized vehicles may include road traffic participation targets such as bicycles and electric bicycles.

[0023] In this embodiment, the lightweight YOLO target detection network includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The C2f-Fb lightweight feature fusion module is used for feature splitting, lightweight feature extraction, and branch feature fusion in this embodiment. The ADown lightweight downsampling module is used for pooling and dual-branch convolution processing in this embodiment. The DySample dynamic upsampling module is used to generate sampling offsets and perform dynamic resampling in this embodiment.

[0024] Specifically, the backbone network is used to extract features from the input intelligent driving scene image to obtain initial feature maps at multiple scales; the neck network is used to perform multi-scale feature fusion on the initial feature maps at multiple scales to obtain fused feature maps at multiple scales; the detection head is used to detect the fused feature maps at multiple scales and output traffic target detection results. The traffic target detection results include the category and location of the traffic target, and in some embodiments, the traffic target detection results may also include the confidence level of the traffic target.

[0025] In this embodiment, the C2f-Fb lightweight feature fusion module is used to perform feature splitting on the input feature map in the neck network, and then perform lightweight feature extraction on at least one branch feature after splitting through the FasterBlock unit, and then fuse it with the features of the remaining branches; the ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map during the downsampling process of the neck network, and then concatenate the dual-branch output features according to the channel dimension to obtain a downsampled feature map for multi-scale feature fusion; the DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map during the upsampling process of the neck network, and determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for multi-scale feature fusion.

[0026] Therefore, this embodiment introduces a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module into the neck network of the lightweight YOLO target detection network. This enables the network to maintain its ability to detect multi-scale traffic targets in intelligent driving scenarios while reducing the number of parameters and computational load, thereby improving the applicability of real-time target detection on the vehicle side.

[0027] Combination Figure 1 As shown, the lightweight YOLO target detection network in this embodiment includes a backbone network, a neck network, and a detection head. After the intelligent driving scene image is input into the lightweight YOLO target detection network, it is processed sequentially through the backbone network, neck network, and detection head, ultimately outputting the traffic target detection result.

[0028] Specifically, the backbone network receives images of intelligent driving scenarios and extracts features from these images to obtain initial feature maps at multiple scales. These initial feature maps at multiple scales can include feature maps with different spatial resolutions and semantic levels. Higher-resolution initial feature maps retain more edge and positional information, while lower-resolution initial feature maps represent stronger semantic information. By outputting initial feature maps at multiple scales, basic features can be provided for subsequent multi-scale detection of traffic targets.

[0029] The neck network receives initial feature maps at multiple scales output from the backbone network and performs multi-scale feature fusion on these initial feature maps to obtain a fused feature map at multiple scales. In this embodiment, the neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The C2f-Fb lightweight feature fusion module reduces redundant computation and enhances branch feature fusion capabilities during feature fusion; the ADown lightweight downsampling module generates downsampled feature maps for multi-scale feature fusion during downsampling; and the DySample dynamic upsampling module generates high-resolution feature maps for multi-scale feature fusion during upsampling.

[0030] It should be noted that, Figure 1 The C2f-Fb lightweight feature fusion module, ADown lightweight downsampling module, and DySample dynamic upsampling module shown in the diagram represent the core processing structures involved in multi-scale feature fusion in the neck network. It is not limited to these three modules operating in only one manner. Figure 1 The unique connection order setting is shown in the diagram. In the actual network structure, the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module can be set at different feature fusion nodes of the neck network according to the fusion relationship between feature maps of different scales.

[0031] The detection head receives fused feature maps of multiple scales output by the neck network and performs target classification prediction and target location regression on the fused feature maps of different scales to obtain traffic target detection results. The traffic target detection results include the category and location of the traffic target; in some embodiments, the traffic target detection results also include the confidence level of the traffic target. By detecting fused feature maps of multiple scales, the detection head can meet the detection needs of traffic targets of different sizes.

[0032] In one optional implementation, the lightweight YOLO object detection network can be an improved version of the YOLOv8 series object detection networks. For example, based on the BSD-YOLO object detection model, the feature fusion structure, downsampling structure, and upsampling structure in its neck network can be lightweighted and improved, making the neck network include a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. This reduces the number of parameters and computational cost of the neck network without changing the basic input-output format of the object detection network, and improves its real-time inference applicability on automotive embedded computing devices.

[0033] Combination Figure 2As shown, the intelligent driving lightweight target detection method provided in this embodiment may include the following steps.

[0034] S101, acquires images of intelligent driving scenarios.

[0035] Specifically, intelligent driving scene images can be acquired through image acquisition devices installed on the vehicle. These image acquisition devices can be in-vehicle cameras, road environment perception cameras, or other devices capable of acquiring images of the road environment surrounding the vehicle. The intelligent driving scene images can be images of the road in front of the vehicle, images of the road to the side of the vehicle, images of intersections, images of highways, images of urban roads, or images of parking scenes.

[0036] In one alternative implementation, the intelligent driving scene image can be preprocessed before being input into the lightweight YOLO object detection network. The preprocessing may include at least one of image resizing, pixel value normalization, color space conversion, and format conversion, to ensure that the preprocessed intelligent driving scene image meets the input requirements of the lightweight YOLO object detection network.

[0037] S102, input the intelligent driving scene image into the lightweight YOLO target detection network.

[0038] Specifically, the acquired intelligent driving scene images are input into the trained lightweight YOLO object detection network. The lightweight YOLO object detection network includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module.

[0039] S103, the backbone network is used to extract features from the intelligent driving scene image to obtain initial feature maps at multiple scales.

[0040] Specifically, the backbone network extracts features layer by layer from the input intelligent driving scene image and outputs initial feature maps at multiple scales at different network depths. These initial feature maps at multiple scales have different spatial resolutions and semantic representation capabilities, and are used to represent road backgrounds, traffic target edges, target textures, and target semantic information in the intelligent driving scene image.

[0041] For example, a higher resolution initial feature map can retain more target edge and location information, making it suitable for detecting small or distant traffic targets; a lower resolution initial feature map can represent stronger semantic information, making it suitable for recognizing larger traffic targets or traffic targets in complex road environments.

[0042] S104, Multi-scale feature fusion is performed on the initial feature maps of multiple scales through the neck network to obtain a fused feature map of multiple scales.

[0043] Specifically, the neck network receives initial feature maps at multiple scales output by the backbone network and, through feature fusion, downsampling, and upsampling, transfers and fuses spatial and semantic information between feature maps at different scales to obtain fused feature maps at multiple scales. In this embodiment, the neck network introduces a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module during the multi-scale feature fusion process.

[0044] Among them, such as Figure 2 As shown in S1041, the C2f-Fb lightweight feature fusion module performs feature splitting and lightweight feature extraction on the input feature map, and then performs branch feature fusion. Specifically, the C2f-Fb lightweight feature fusion module splits the input feature map, and then uses the FasterBlock unit to extract lightweight features from at least one branch feature after splitting, and then fuses it with the remaining branch features.

[0045] like Figure 2 As shown in S1042, the ADown lightweight downsampling module performs pooling and dual-branch convolution processing on the input feature map to obtain a downsampled feature map. Specifically, after pooling the input feature map, the ADown lightweight downsampling module divides the pooled feature map into two branches, and performs convolution and pooling downsampling processing on each branch. Then, the output features of the two branches are concatenated according to the channel dimension to obtain a downsampled feature map for multi-scale feature fusion.

[0046] like Figure 2 As shown in S1043, the DySample dynamic upsampling module generates a sampling offset based on the input feature map and performs resampling to obtain a high-resolution feature map. Specifically, the DySample dynamic upsampling module generates a sampling offset based on the input feature map, determines a dynamic sampling position based on the sampling offset and the basic sampling grid, and resamples the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for multi-scale feature fusion.

[0047] It should be noted that S1041, S1042, and S1043 are used to illustrate different feature processing methods involved in the multi-scale feature fusion process of the neck network, and do not limit the three to following the prescribed methods. Figure 2The numbering order shown in the diagram is executed sequentially along a single path. In the actual network structure, the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module can be set at different feature fusion nodes in the neck network to complete the fusion, downsampling, and upsampling processing between feature maps of different scales.

[0048] The following is combined with Figures 3 to 7 This paper further explains the C2f-Fb lightweight feature fusion module, FasterBlock unit, partial channel convolution, ADown lightweight downsampling module, and DySample dynamic upsampling module involved in multi-scale feature fusion in the neck network of S104.

[0049] Combination Figure 3 As shown, the C2f-Fb lightweight feature fusion module in this embodiment includes a feature splitting unit, at least one FasterBlock unit, and a feature fusion unit. The C2f-Fb lightweight feature fusion module is used to perform feature splitting, lightweight feature extraction, and branch feature fusion on the input feature map during the multi-scale feature fusion process of the neck network, obtaining a fused output feature map.

[0050] Specifically, the feature splitting unit receives the input feature map and divides it into retained branch features and processed branch features. The retained branch features are used to retain the original feature information in the input feature map, while the processed branch features are input to the FasterBlock unit for further lightweight feature extraction.

[0051] After lightweight feature extraction by at least one FasterBlock unit, the processed branch features are obtained. The FasterBlock unit processes the branch features using partial-channel convolutions and 1×1 convolutions. Partial-channel convolutions reduce the computational cost of spatial convolutions, while 1×1 convolutions are used for channel-dimensional feature fusion. The specific structure of the FasterBlock unit will be discussed later. Figure 4 and Figure 5 Further explanation.

[0052] The feature fusion unit is used to fuse the branch features processed by the FasterBlock unit with the retained branch features to obtain a fused output feature map. The fusion can include channel-dimensional concatenation, convolutional fusion, or other methods capable of fusing branch features. In one specific embodiment, the feature fusion unit can concatenate the processed branch features and the retained branch features along the channel dimension, and then obtain the fused output feature map through convolutional processing.

[0053] In one optional implementation, the C2f-Fb lightweight feature fusion module can be an improved feature fusion module based on the C2f structure of the YOLOv8 series object detection networks. Specifically, the feature splitting and feature fusion framework in the C2f structure can be retained, and the bottleneck unit in the original C2f structure can be replaced with a FasterBlock unit, thereby obtaining the C2f-Fb lightweight feature fusion module. In this way, the C2f-Fb lightweight feature fusion module can reduce redundant computation in the neck network feature fusion process while maintaining the branch feature fusion capability.

[0054] It should be noted that, Figure 3 The example illustrates multiple FasterBlock units, but this example is not intended to limit the specific number of FasterBlock units. In other implementations, the number of FasterBlock units can be set to one or more depending on the network depth of the lightweight YOLO object detection network, the number of channels in the input feature map, computational resource constraints, and detection accuracy requirements.

[0055] This embodiment introduces a FasterBlock unit into the C2f-Fb lightweight feature fusion module, which reduces the computational overhead caused by performing spatial convolution on all channels when the neck network performs multi-scale feature fusion. At the same time, by preserving branch features and processing the fusion of branch features, the impact of lightweight processing on the expressive ability of traffic target features is reduced.

[0056] Combination Figure 4 As shown, the FasterBlock unit in this embodiment includes a 3×3 partial channel convolution, a channel feature merging unit, and two layers of 1×1 ordinary convolutions arranged sequentially. After the input feature map enters the FasterBlock unit, it first undergoes spatial convolution processing on some continuous channels of the input feature map through the 3×3 partial channel convolution, and the remaining channels that do not participate in the spatial convolution are treated as directly connected channels and skipped from the 3×3 partial channel convolution. Subsequently, the channel feature merging unit merges the channel features processed by the 3×3 partial channel convolution with the channel features of the directly connected channels. The merged channel features are then sequentially input into two layers of 1×1 ordinary convolutions, where channel dimensions are fused to obtain the output feature map of the FasterBlock unit.

[0057] After completing the 3×3 partial channel convolution processing, the channel features processed by the 3×3 partial channel convolution and the channel features of the directly connected channels are jointly fed into two subsequent 1×1 ordinary convolution layers. These two 1×1 ordinary convolution layers are used to fuse the channel features processed by the 3×3 partial channel convolution and the channel features of the directly connected channels in terms of channel dimension, thereby enhancing the information interaction between different channels and outputting the processing result of the FasterBlock unit.

[0058] It should be noted that the "direct-connected channels" refer to the remaining channels that do not participate in spatial convolution, which are not processed by 3×3 spatial convolution but are included as part of the output features of the 3×3 partial channel convolution. They are then combined with the channel features processed by spatial convolution and entered into the subsequent 1×1 ordinary convolution for channel dimension fusion. This does not mean that these channels bypass all subsequent processing in the FasterBlock unit and are directly used as the final output.

[0059] In one optional implementation, the number of the partially continuous channels can be determined based on the total number of channels in the input feature map and a preset channel partitioning ratio. For example, a portion of the continuously arranged channels from all channels of the input feature map can be selected as partially continuous channels participating in spatial convolution, while the remaining unselected channels are treated as directly connected channels. The preset channel partitioning ratio can be set according to the computational requirements, detection accuracy requirements, and vehicle-mounted deployment conditions of the lightweight YOLO object detection network.

[0060] This embodiment employs 3×3 partial channel convolution in the FasterBlock unit to extract spatial features from only some continuous channels, and completes channel dimension fusion through two layers of 1×1 ordinary convolution. This allows the FasterBlock unit to maintain effective channel information of the input feature map while reducing the computational cost of spatial convolution, thereby improving the lightweight feature expression capability of the C2f-Fb lightweight feature fusion module in the neck network.

[0061] Combination Figure 5 As shown, the 3×3 partial channel convolution can be achieved by first dividing the input feature map along its channel dimensions to obtain convolutional channel groups and directly connected channel groups. The convolutional channel groups include the continuous channels in the input feature map that participate in spatial convolution, and the directly connected channel groups include the remaining channels in the input feature map that do not participate in spatial convolution.

[0062] For the convolutional channel group, a 3×3 spatial convolution is performed to extract local spatial features from the convolutional channel group. For the directly connected channel group, no 3×3 spatial convolution is performed; instead, the features are directly passed within the 3×3 partial channel convolution. Subsequently, the features of the convolutional channel group after the 3×3 spatial convolution and the directly passed features of the directly connected channel group are merged to obtain the output feature map of the 3×3 partial channel convolution.

[0063] It should be noted that the direct transmission of the directly connected channel group means that the directly connected channel group skips the 3×3 spatial convolution processing and participates in the channel dimension fusion of the subsequent two layers of 1×1 ordinary convolution as part of the output features of the 3×3 partial channel convolution; it does not mean that the directly connected channel group bypasses the subsequent 1×1 ordinary convolution in the FasterBlock unit and directly becomes the final output of the FasterBlock unit.

[0064] By employing the above method, partial channel convolution performs spatial convolution only on a subset of channels of the input feature map, rather than on all channels, thus significantly reducing the computational cost of spatial convolution. Simultaneously, since the remaining channels not involved in spatial convolution participate as directly connected channels in subsequent channel dimensionality fusion, effective feature information in the input feature map can be preserved while reducing computational cost.

[0065] In one alternative implementation, the number of channels participating in spatial convolution can be determined based on the total number of channels in the input feature map and a preset channel partitioning ratio. The preset channel partitioning ratio can be set according to the model's lightweight requirements, the computing power limitations of in-vehicle embedded computing devices, and the accuracy requirements for traffic target detection.

[0066] Combination Figure 6 As shown, the ADown lightweight downsampling module in this embodiment is used to perform pooling and dual-branch convolution processing on the input feature map during the downsampling process of the neck network, and to concatenate the dual-branch output features according to the channel dimension to obtain a downsampled feature map for multi-scale feature fusion.

[0067] Specifically, the ADown lightweight downsampling module first performs 2×2 average pooling with a stride of 1 and no padding on the input feature map to obtain a pooled feature map. By performing average pooling before dual-branch processing, a smoother feature input can be provided for subsequent branch processing while preserving the local statistical information of the input feature map.

[0068] Then, the ADown lightweight downsampling module divides the pooled feature map into a first branch feature and a second branch feature. The first branch feature and the second branch feature are respectively entered into different downsampling processing paths to obtain downsampling-related features from different processing methods.

[0069] For the first branch feature, the ADown lightweight downsampling module performs a 3×3 convolution on the first branch feature and adjusts the number of channels to half of the target total number of channels. The target total number of channels is the total number of channels required for the downsampled feature map output by the ADown lightweight downsampling module. The first branch extracts local spatial features through 3×3 convolution processing and outputs the processed features of the first branch.

[0070] For the second branch features, the ADown lightweight downsampling module performs 1×1 convolution and max-pooling downsampling on the second branch features to obtain the processed second branch features. In one specific implementation, the max-pooling downsampling uses 3×3 max-pooling with a stride of 2 and padding of 1. The 1×1 convolution is used to adjust the channel representation of the second branch features, and the max-pooling downsampling is used to preserve feature information with strong local responses during the downsampling process.

[0071] Subsequently, the ADown lightweight downsampling module concatenates the processed first branch features and the processed second branch features according to the channel dimension to obtain the downsampled feature map. This downsampled feature map serves as one of the input features in the multi-scale feature fusion process of the neck network, and is used to fuse with feature maps of other scales.

[0072] In this way, the ADown lightweight downsampling module does not simply use conventional convolution to perform downsampling. Instead, it generates a downsampled feature map through average pooling, first-branch convolution processing, second-branch 1×1 convolution and max-pooling downsampling processing, and channel-dimensional concatenation. This allows for feature map size adjustment while reducing redundant computation in the downsampling process and improving the ability of the downsampled feature map to retain effective features of traffic targets.

[0073] In one alternative implementation, the ADown lightweight downsampling module can be placed at the feature transfer node in the neck network that requires scale reduction, replacing the conventional convolutional downsampling structure. By introducing the ADown lightweight downsampling module into the neck network, the computational overhead in the multi-scale feature fusion process can be reduced, making the lightweight YOLO object detection network more suitable for deployment in automotive embedded computing devices.

[0074] Combination Figure 7 As shown, the DySample dynamic upsampling module in this embodiment is used to generate a sampling offset based on the input feature map during the upsampling process of the neck network, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and then resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for multi-scale feature fusion.

[0075] Specifically, the DySample dynamic upsampling module includes a sampling point generator. After the input feature map is input to the sampling point generator, the generator generates sampling offsets through a linear mapping. The process of generating the sampling offsets can be represented as follows:

[0076] in, Indicates input features, It is a linear mapping function. This is the initial sampling offset.

[0077] To enhance the stability of the sampling range, the DySample dynamic upsampling module introduces a dynamic range constraint factor during offset generation. Specifically, the sampling point generator may include a range factor branch and an offset prediction branch. The range factor branch is used to linearly map the input feature map and normalize it using the Sigmoid function (a sigmoid activation function). The offset prediction branch is used to linearly map the input feature map. In one specific implementation, the preset scaling factor is 0.5.

[0078] The sampling offset after introducing dynamic range constraints can be expressed as:

[0079] in, This represents the dynamic range constraint factor obtained from the range factor branch. This represents the offset prediction amount obtained from the offset prediction branch.

[0080] Then, the DySample dynamic upsampling module adds the constrained sampling offset to the base sampling grid to obtain the dynamic sampling position. The base sampling grid is used to represent the base sampling coordinates corresponding to the upsampling process, and the constrained sampling offset is used to adjust the position based on the base sampling coordinates.

[0081]

[0082] in, Represents the basic sampling grid. This represents the sampling offset after constraints. Indicates the dynamic sampling position.

[0083] Finally, the DySample dynamic upsampling module resamples the input feature map through a grid-based bilinear interpolation resampling operation to generate the high-resolution feature map.

[0084]

[0085] in, Represents high-resolution feature maps, Indicates the input feature map, Indicates the dynamic sampling position. This indicates a bilinear interpolation resampling operation based on the sampling grid.

[0086] It should be noted that during the resampling process, the input feature map is used both to generate the sampling offset and as the object to be resampled; the dynamic sampling position is used to determine the location for feature sampling from the input feature map. Therefore, the resampling process of the DySample dynamic upsampling module does not generate a high-resolution feature map independently based solely on the dynamic sampling position, but rather reads the corresponding features from the input feature map based on the dynamic sampling position and generates a high-resolution feature map through bilinear interpolation.

[0087] This embodiment uses the DySample dynamic upsampling module to adaptively generate dynamic sampling positions based on the input feature map. Compared with the static upsampling method that uses fixed interpolation rules, it can make the upsampling process more adaptable to the traffic target morphology and edge features in different intelligent driving scenario images, thereby improving the neck network's ability to recover multi-scale traffic target features.

[0088] In one alternative implementation, the DySample dynamic upsampling module can be placed at the feature transfer node in the neck network that requires scale upsampling, replacing the conventional static upsampling structure. By introducing the DySample dynamic upsampling module into the neck network, the ability to represent detailed information in high-resolution feature maps can be improved while maintaining a lightweight structure, making the lightweight YOLO object detection network more suitable for traffic object detection in intelligent driving scenarios.

[0089] Through the processing of the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module, the neck network can perform lightweight multi-scale feature fusion on the initial feature maps of multiple scales output by the backbone network, ultimately forming a multi-scale fused feature map. This multi-scale fused feature map serves as input to the detection head for subsequent traffic target classification prediction and location regression.

[0090] S105, the detection head detects the fused feature map of multiple scales output by the neck network and outputs the traffic target detection result.

[0091] Specifically, the detection head receives fused feature maps at multiple scales output by the neck network, and performs target classification prediction and target location regression on the fused feature maps at different scales respectively to obtain the category and location of the traffic target. In an optional embodiment, the detection head can also output the confidence level of the traffic target. The traffic target may include at least one of vehicles, pedestrians, and non-motorized vehicles.

[0092] Thus, the lightweight YOLO object detection network completes the reasoning process from intelligent driving scene image input, backbone network feature extraction, neck network multi-scale feature fusion, to the detection head outputting traffic object detection results.

[0093] The above embodiments illustrate the process of traffic object detection in intelligent driving scene images during the inference phase of the trained lightweight YOLO object detection network. To obtain the trained lightweight YOLO object detection network, in one optional implementation, the lightweight YOLO object detection network can be trained first, as detailed below.

[0094] In one alternative implementation, the lightweight YOLO object detection network can be trained before being deployed to an in-vehicle embedded computing device to obtain a trained lightweight YOLO object detection network.

[0095] Specifically, a training dataset can be constructed. The training dataset includes multiple images of intelligent driving scenarios and corresponding traffic target annotation information. The traffic target annotation information may include category labels and location bounding boxes for the traffic targets. The traffic targets may include at least one of vehicles, pedestrians, and non-motorized vehicles.

[0096] During training, intelligent driving scene images from the training dataset are input into a lightweight YOLO object detection network. The backbone network extracts features from the intelligent driving scene images to obtain initial feature maps at multiple scales. The neck network performs multi-scale feature fusion on the initial feature maps at multiple scales to obtain fused feature maps at multiple scales. The detection head detects the fused feature maps at multiple scales to obtain the predicted detection results.

[0097] The predicted detection results may include the predicted target category, the predicted target location, and the predicted confidence level. Based on the predicted detection results and the corresponding traffic target annotation information, a target detection loss is calculated. The target detection loss may include at least one of classification loss, location regression loss, and confidence level loss.

[0098] After obtaining the target detection loss, the network parameters in the lightweight YOLO target detection network are updated based on the target detection loss. The network parameters include parameters in the backbone network, parameters in the neck network, and parameters in the detection head. Specifically, the parameters in the neck network include parameters from the C2f-Fb lightweight feature fusion module, parameters from the ADown lightweight downsampling module, and parameters from the DySample dynamic upsampling module.

[0099] Furthermore, in the DySample dynamic upsampling module, the linear mapping function parameters in the range factor branch and the linear mapping function parameters in the offset prediction branch can be trained together with the lightweight YOLO object detection network. Thus, the DySample dynamic upsampling module can learn to generate the sampling range factor and offset prediction based on the input feature map, and further generate the constrained sampling offset.

[0100] After multiple rounds of iterative training, when the object detection loss meets a preset convergence condition, or when the number of training rounds reaches a preset threshold, a lightweight YOLO object detection network is obtained. This lightweight YOLO object detection network is used to perform traffic object detection on intelligent driving scene images.

[0101] In one alternative implementation, the original YOLOv8 series object detection network or the BSD-YOLO object detection model can be used as the base network. A C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module are introduced into the neck network of the base network to obtain the lightweight YOLO object detection network to be trained. After training, the lightweight YOLO object detection network can reduce the number of network parameters and computational load while maintaining traffic object detection capabilities.

[0102] It should be noted that this embodiment does not limit the specific source of the training dataset, the number of training rounds, the learning rate, the optimizer type, and the batch size. Those skilled in the art can set these parameters according to the data scale of the intelligent driving scene images, the deployment requirements of the vehicle-mounted embedded computing device, and the detection accuracy requirements.

[0103] In one alternative implementation, the trained lightweight YOLO object detection network can be deployed to an in-vehicle embedded computing device, and the in-vehicle embedded computing device can be used to perform object detection on real-time acquired intelligent driving scene images and output traffic object detection results.

[0104] Specifically, the in-vehicle embedded computing device can be installed in an intelligent driving vehicle or an assisted driving vehicle and communicate with an image acquisition device. The image acquisition device can be an in-vehicle camera, a road environment perception camera, or other devices capable of acquiring images of the road environment surrounding the vehicle. During vehicle operation, the image acquisition device acquires real-time images of the intelligent driving scene surrounding the vehicle and sends these images to the in-vehicle embedded computing device.

[0105] After receiving an image of a smart driving scene, the in-vehicle embedded computing device calls a trained lightweight YOLO object detection network to process the image. Specifically, the lightweight YOLO object detection network extracts initial feature maps at multiple scales through the backbone network, performs multi-scale feature fusion on the initial feature maps at multiple scales through the neck network, and outputs traffic object detection results through the detection head.

[0106] The traffic target detection result may include the category and location of at least one traffic target among vehicles, pedestrians, and non-motorized vehicles. In an optional embodiment, the traffic target detection result may further include the confidence level of the traffic target. The location may be represented by a target detection bounding box, target center point coordinates, target width and height information, or other information that can characterize the area where the traffic target is located in the intelligent driving scene image.

[0107] After obtaining the traffic target detection results, the onboard embedded computing device can send these results to the environmental perception module, path planning module, obstacle avoidance module, or vehicle control module of the intelligent driving system. The intelligent driving system can then determine the location and category of traffic participants around the vehicle based on the traffic target detection results, and further perform processing such as path planning, driving decisions, obstacle avoidance control, or early warning prompts.

[0108] Since automotive embedded computing devices are typically limited by computing power, power consumption, storage space, and real-time requirements, directly deploying object detection networks with a large number of parameters and high floating-point operation requirements can easily lead to high inference latency or insufficient real-time detection capabilities. This embodiment introduces a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module into the neck network of a lightweight YOLO object detection network. This reduces the computational overhead of the neck network during multi-scale feature fusion, thereby improving the deployment applicability of the trained lightweight YOLO object detection network on automotive embedded computing devices.

[0109] In one specific implementation, the in-vehicle embedded computing device can input frame-by-frame trained lightweight YOLO object detection network with continuously acquired intelligent driving scene images to obtain the traffic object detection result corresponding to each frame of the intelligent driving scene image. By continuously outputting the traffic object detection results, real-time environmental perception information can be provided to the intelligent driving vehicle.

[0110] It should be noted that the vehicle-mounted embedded computing device in this embodiment may include a processor, a memory, and a communication interface. The memory may store the network parameters and computer program of the trained lightweight YOLO object detection network. When the processor executes the computer program, it realizes traffic object detection on intelligent driving scene images. The specific hardware type of the vehicle-mounted embedded computing device is not limited and can be selected according to the vehicle platform, computing power requirements, and power consumption limitations.

[0111] In one optional implementation, to verify the detection performance and lightweight effect of the lightweight YOLO object detection network provided in this embodiment, the original object detection model can be used as a control model, and the lightweight YOLO object detection network in this embodiment can be used as an improved model. The performance can be compared under the same training dataset, the same test dataset, and the same test hardware environment.

[0112] Specifically, the original BSD-YOLO object detection model can be selected as the control model, and the lightweight YOLO object detection network obtained by introducing the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module on the basis of the original BSD-YOLO object detection model can be selected as the improved model. The control model and the improved model are trained respectively, and the number of parameters, floating-point operation, detection accuracy, and inference frame rate of the two are statistically analyzed under the same test conditions.

[0113] In one specific implementation, based on experimental results, the lightweight YOLO object detection network in this embodiment, compared to the original BSD-YOLO object detection model, reduces the number of parameters by approximately 30%, decreases floating-point computation by approximately 35%, significantly improves the device inference frame rate, and exhibits only minor fluctuations in overall detection accuracy. Therefore, this embodiment can maintain the ability to detect traffic targets in intelligent driving scenarios while reducing the number of network parameters and computational load.

[0114] From the perspective of module function, the C2f-Fb lightweight feature fusion module reduces the spatial convolution computation of the neck network in the feature fusion process by using partial channel convolution in the FasterBlock unit; the ADown lightweight downsampling module generates downsampled feature maps by average pooling, dual-branch convolution processing and channel dimension concatenation, reducing the redundant computation caused by conventional convolution downsampling; the DySample dynamic upsampling module adaptively generates dynamic sampling positions based on the input feature map and generates high-resolution feature maps by resampling through bilinear interpolation, so as to improve the ability to recover traffic target edges and detailed features during the upsampling process.

[0115] Therefore, this embodiment does not simply reduce the number of network layers or compress the number of channels, but rather makes lightweight improvements to the three processing stages of the neck network: feature fusion, downsampling, and upsampling. Through the synergistic effect of the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module, the lightweight YOLO object detection network can maintain the detection performance of multi-scale traffic targets in intelligent driving scenarios while adapting to the computing power limitations of in-vehicle embedded computing devices.

[0116] It should be noted that the above experimental data is used to illustrate the technical effects achievable in this embodiment and is not intended to limit the scope of protection of this application. In other embodiments, the specific number of parameters, floating-point operations, detection accuracy, and inference frame rate may vary depending on the training dataset, test dataset, input image size, hardware platform, training parameters, and model configuration.

[0117] Combination Figure 8 As shown, this embodiment also provides a lightweight target detection device for intelligent driving. The lightweight target detection device for intelligent driving can be used to execute the lightweight target detection method for intelligent driving described in any of the above embodiments. The device includes an image acquisition module, a network input module, a feature extraction module, a multi-scale feature fusion module, and a detection output module.

[0118] The image acquisition module is used to acquire images of the intelligent driving scene. Specifically, the image acquisition module can acquire intelligent driving scene images from vehicle-mounted cameras, road environment perception cameras, or other image acquisition devices. The intelligent driving scene images can be road environment images collected in real time during vehicle operation.

[0119] The network input module is used to input the intelligent driving scene image into a lightweight YOLO object detection network. The lightweight YOLO object detection network includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. In an optional embodiment, the network input module can also be used to resize, normalize, or convert the format of the intelligent driving scene image to meet the input requirements of the lightweight YOLO object detection network.

[0120] The feature extraction module is used to extract features from the intelligent driving scene image through the backbone network to obtain initial feature maps at multiple scales. These initial feature maps at different scales have different spatial resolutions and semantic levels, providing basic features for multi-scale feature fusion in the neck network.

[0121] The multi-scale feature fusion module is used to perform multi-scale feature fusion on the initial feature maps of multiple scales through the neck network to obtain a fused feature map of multiple scales. Specifically, the C2f-Fb lightweight feature fusion module is used to perform feature splitting on the input feature map, and after extracting lightweight features from at least one branch feature through a FasterBlock unit, it is fused with the remaining branch features. The ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map, and concatenate the dual-branch output features according to the channel dimension to obtain a downsampled feature map for the multi-scale feature fusion. The DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for the multi-scale feature fusion.

[0122] The detection output module is used to detect the multi-scale fused feature map output by the neck network through the detection head, and output traffic target detection results. The traffic target detection results include the category and location of the traffic target. In an optional embodiment, the traffic target detection results may further include the confidence level of the traffic target. The traffic target may include at least one of vehicles, pedestrians, and non-motorized vehicles.

[0123] In one optional implementation, the image acquisition module, network input module, feature extraction module, multi-scale feature fusion module, and detection output module can be implemented by a processor executing program instructions stored in memory. That is, the intelligent driving lightweight target detection device can be a software module-based device, with each module corresponding to and executing the relevant steps in the aforementioned intelligent driving lightweight target detection method.

[0124] In another optional implementation, the image acquisition module, network input module, feature extraction module, multi-scale feature fusion module, and detection output module can also be implemented by hardware circuitry, or by a combination of hardware circuitry and software programs. This embodiment does not limit the specific implementation form of each module.

[0125] It should be noted that the intelligent driving lightweight target detection device in this embodiment and the intelligent driving lightweight target detection method described above belong to the same inventive concept. The specific processing procedures of each module in the device embodiment can be found in the descriptions of the corresponding steps in the method embodiment described above, and will not be repeated here.

[0126] In one alternative embodiment, this embodiment also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, it implements the intelligent driving lightweight target detection method described in any of the above embodiments.

[0127] In one alternative implementation, the electronic device may be an in-vehicle embedded computing device, an in-vehicle intelligent driving controller, an edge computing device, or other computing devices with image processing capabilities.

[0128] In an optional embodiment, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the intelligent driving lightweight target detection method described in any of the above embodiments.

[0129] It should be noted that the electronic device and computer-readable storage medium in this embodiment belong to the same inventive concept as the above-mentioned intelligent driving lightweight target detection method. The specific implementation process can be found in the aforementioned method embodiment and device embodiment, and will not be repeated here.

[0130] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit the scope of protection of this application. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical features in the above embodiments can be combined, substituted, or modified without departing from the essence of the technical solutions of this application, and such combinations, substitutions, or modifications should still fall within the scope of protection of this application.

[0131] It should be noted that the names YOLO, C2f-Fb, FasterBlock, ADown, DySample, and Grid Sample used in this application refer to the corresponding object detection networks, network modules, network units, or resampling operations. The specific structures and processing flows of the C2f-Fb lightweight feature fusion module, the ADown lightweight downsampling module, and the DySample dynamic upsampling module are as defined in the foregoing embodiments and should not be interpreted narrowly based solely on their names.

[0132] It should also be noted that the step numbers in the embodiments of this application are only for ease of description and do not limit the steps to be executed sequentially in all implementations. For the C2f-Fb lightweight feature fusion module, ADown lightweight downsampling module, and DySample dynamic upsampling module in the neck network, they can be set at different feature fusion nodes of the neck network according to the fusion relationship between feature maps of different scales, as long as the aforementioned multi-scale feature fusion, downsampling, and upsampling processing can be achieved.

[0133] The device modules in the embodiments of this application can be merged, split, or renamed according to actual needs. As long as they can perform the functions and processes corresponding to the above method embodiments, they are all equivalent implementations of this application.

Claims

1. A lightweight target detection method for intelligent driving, characterized in that, include: Acquire images of intelligent driving scenarios; The intelligent driving scene image is input into a lightweight YOLO target detection network, which includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The backbone network is used to extract features from the intelligent driving scene images to obtain initial feature maps at multiple scales. The neck network is used to perform multi-scale feature fusion on the initial feature maps of the multiple scales to obtain a fused feature map of the multiple scales. The C2f-Fb lightweight feature fusion module is used to split the input feature map into features and extract lightweight features from at least one branch feature through the FasterBlock unit, and then fuse it with the features of the remaining branches. The FasterBlock unit includes partial channel convolution and 1×1 convolution. The partial channel convolution is used to extract spatial features from a portion of the input feature map and allows the remaining channels that do not participate in spatial feature extraction to skip the partial channel convolution as directly connected channels. The 1×1 convolution is used to receive the channel features processed by the partial channel convolution and the channel features of the directly connected channels, and perform channel dimension feature fusion. During the downsampling process of the neck network, the ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map, and to concatenate the dual-branch output features according to the channel dimension to obtain the downsampled feature map used for the multi-scale feature fusion. During the upsampling process of the neck network, the DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for the multi-scale feature fusion. The detection head detects the fused feature maps of multiple scales output by the neck network and outputs traffic target detection results, which include the category and location of the traffic targets.

2. The intelligent driving lightweight target detection method according to claim 1, characterized in that, The C2f-Fb lightweight feature fusion module includes a feature splitting unit, at least one FasterBlock unit, and a feature fusion unit connected in sequence. The feature splitting unit is used to divide the input feature map into retained branch features and processing branch features. The processing branch features are extracted by the at least one FasterBlock unit in a lightweight manner, and then fused with the retained branch features by the feature fusion unit to obtain a fused output feature map.

3. The intelligent driving lightweight target detection method according to claim 2, characterized in that, The FasterBlock unit comprises a 3×3 partial channel convolution layer and two 1×1 ordinary convolution layers arranged sequentially. The 3×3 partial channel convolution layer is used to perform spatial convolution only on a portion of the continuous channels of the input feature map, and allows the remaining channels that do not participate in the spatial convolution to skip the 3×3 partial channel convolution layer as directly connected channels. The two 1×1 ordinary convolution layers are used to receive the channel features processed by the 3×3 partial channel convolution layer and the channel features of the directly connected channels, and to fuse the two in terms of channel dimension.

4. The intelligent driving lightweight target detection method according to claim 1, characterized in that, The ADown lightweight downsampling module performs 2×2 average pooling with a stride of 1 and no padding on the input feature map to obtain a pooled feature map. The pooled feature map is then divided into a first branch feature and a second branch feature. The first branch feature is subjected to 3×3 convolution, and the number of channels is adjusted to half of the target total number of channels. The second branch feature is subjected to 1×1 convolution and max pooling downsampling, where the max pooling downsampling uses 3×3 max pooling with a stride of 2 and padding of 1. The processed first branch feature and second branch feature are concatenated according to the channel dimension to obtain the downsampled feature map.

5. The intelligent driving lightweight target detection method according to claim 1, characterized in that, The DySample dynamic upsampling module generates a sampling offset based on the input feature map, including: inputting the input feature map into a sampling point generator, the sampling point generator including a range factor branch and an offset prediction branch; performing a linear mapping on the input feature map through the range factor branch and normalizing it through a sigmoid activation function to obtain a sampling range factor; performing a linear mapping on the input feature map through the offset prediction branch to obtain an offset prediction; and obtaining a constrained sampling offset based on the sampling range factor and the offset prediction.

6. The intelligent driving lightweight target detection method according to claim 5, characterized in that, The step of obtaining the constrained sampling offset based on the sampling range factor and the offset prediction includes: multiplying the sampling range factor, the offset prediction, and a preset scaling factor to obtain the constrained sampling offset; wherein the preset scaling factor is 0.

5.

7. The intelligent driving lightweight target detection method according to claim 6, characterized in that, The DySample dynamic upsampling module adds the constrained sampling offset to the basic sampling grid to obtain the dynamic sampling position, and resamples the input feature map through a bilinear interpolation resampling operation based on the sampling grid to generate the high-resolution feature map.

8. The intelligent driving lightweight target detection method according to claim 1, characterized in that, The step of detecting multiple scales of fused feature maps output by the neck network through the detection head and outputting traffic target detection results includes: performing target classification prediction and target location regression on the fused feature maps of different scales through the detection head to obtain the category, location and confidence of the traffic target; wherein the traffic target includes at least one of vehicles, pedestrians and non-motorized vehicles.

9. The intelligent driving lightweight target detection method according to claim 1, characterized in that, The trained lightweight YOLO object detection network is deployed to an in-vehicle embedded computing device, and the in-vehicle embedded computing device performs object detection on real-time acquired intelligent driving scene images, and outputs the traffic object detection results.

10. A lightweight target detection device for intelligent driving, characterized in that, include: Image acquisition module, used to acquire images of intelligent driving scenarios; The network input module is used to input the intelligent driving scene image into the lightweight YOLO target detection network. The lightweight YOLO target detection network includes a backbone network, a neck network, and a detection head. The neck network includes a C2f-Fb lightweight feature fusion module, an ADown lightweight downsampling module, and a DySample dynamic upsampling module. The feature extraction module is used to extract features from the intelligent driving scene image through the backbone network to obtain initial feature maps at multiple scales. A multi-scale feature fusion module is used to perform multi-scale feature fusion on the initial feature maps of multiple scales through the neck network to obtain a fused feature map of multiple scales. Specifically, the C2f-Fb lightweight feature fusion module is used to perform feature splitting on the input feature map, and after extracting lightweight features from at least one branch feature through a FasterBlock unit, it is fused with the remaining branch features. The ADown lightweight downsampling module is used to perform pooling and dual-branch convolution processing on the input feature map, and concatenate the dual-branch output features according to the channel dimension to obtain a downsampled feature map for the multi-scale feature fusion. The DySample dynamic upsampling module is used to generate a sampling offset based on the input feature map, determine the dynamic sampling position based on the sampling offset and the basic sampling grid, and resample the input feature map according to the dynamic sampling position to obtain a high-resolution feature map for the multi-scale feature fusion. The detection output module is used to detect the multi-scale fused feature map output by the neck network through the detection head, and output traffic target detection results, which include the category and location of the traffic target.

Images