Salient target detection method of intelligent auxiliary driving system

Abstract

The invention discloses a saliency target detection method for an intelligent auxiliary driving system, and the method comprises the following steps: 1, obtaining a driving scene image based on a driver visual region in real time through a vehicle-mounted image collection device, and carrying out the preprocessing of the image; step 2, performing bottom-layer multi-scale feature extraction, interaction and global enhancement of feature information on the backbone network, and extracting rich multi-scale feature information; step 3, the neck network comprises an AESwinTransform attention module, and a cross-layer feature guide fusion module and a multi-level feature interaction fusion module in the TFG network, and the multi-level features extracted from the backbone network are fused together; step 4, detecting and classifying the feature map; and 5, inputting the adversarial sample and the normal sample into the model in a mixed manner, and performing adversarial training by using the trained model. According to the method, the detection precision of the small saliency target can be improved, the generalization ability and robustness of the model in a complex environment are improved, and the interference of irrelevant scene information is reduced.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a significant target detection method for an intelligent assisted driving system. Background Art

[0002] Technical background of intelligent assisted driving system: Advanced Driver Assistance Systems (ADAS) refers to a system that helps drivers improve driving safety, comfort and convenience by using advanced sensors, control technologies and algorithms. These systems sense the vehicle's surroundings, identify potential dangerous targets, and give drivers prompts and warnings or perform automated control at appropriate times, thereby reducing the driver's burden and the occurrence of traffic accidents. ADAS systems are an important stage in the development of fully automatic driving (unmanned driving).

[0003] The development of intelligent assisted driving technology has gone through several stages, from the earliest passive safety systems (such as seat belts, airbags, etc.) to active safety systems, and then to the current intelligent assisted driving and future fully autonomous driving systems. ADAS is the foundation of autonomous driving technology. Currently, widely used systems such as adaptive cruise control (ACC), lane keeping assist (LKA), automatic emergency braking (AEB), etc. are all primary functions for realizing autonomous driving.

[0004] The realization of intelligent assisted driving systems relies on the combination of multiple technologies, mainly including the following core technologies: (1) Sensor technology: ADAS systems use various sensors (such as cameras, radars, and lidars) to perceive the environment and status around the vehicle. Cameras can identify significant targets such as motor vehicles, obstacles, and pedestrians; radars and lidars are used to detect the distance, speed, and position of obstacles, especially in bad weather and low light conditions. (2) Computer vision and image processing: Relying on the image data collected by the camera, computer vision technology is used to analyze and understand environmental information, such as identifying vehicles, pedestrians, and traffic lights. Significant target detection, object classification, and scene understanding are the core tasks of computer vision technology in ADAS. (3) Sensor fusion (multimodal fusion): In order to provide more accurate and comprehensive environmental perception, ADAS systems usually need to fuse data from different sensors (such as radars, cameras, and lidars). Through the fusion of sensor data, the shortcomings of a single sensor can be compensated and the robustness and accuracy of the system can be improved. (4) Deep learning network: With the development of deep learning technology, many ADAS functions rely on deep learning algorithms, especially deep neural networks (DNNs) to identify and detect target objects. Deep learning algorithms can process large amounts of complex data and continuously improve the accuracy of target recognition and detection through self-learning and training. (5) Control and decision algorithms: The control algorithms in ADAS systems need to make decisions based on perception information, such as adjusting vehicle speed, changing direction, etc. Typical control algorithms include PID control, adaptive control, and predictive control. (6) Vehicle to Everything (V2X) technology: Vehicle to Everything (V2X) technology enables vehicles to exchange information with other vehicles (V2V), road infrastructure (V2I), and pedestrian devices through wireless communication. This communication can expand the vehicle's perception range and enable it to respond to potential dangers in advance.

[0005] With the development of deep learning technology, especially the rise of convolutional neural network (CNN), object detection technology has made significant breakthroughs. CNN has the ability to automatically extract features and can extract more semantic information, greatly improving the accuracy and robustness of object detection.

[0006] R-CNN series: R-CNN (Regions with Convolutional Neural Networks) is a milestone in the application of deep learning to object detection. The R-CNN method extracts candidate regions and then classifies and regresses each region. Subsequent Fast R-CNN and Faster R-CNN have optimized the speed and performance of R-CNN, and gradually become a classic framework for object detection. Faster R-CNN significantly improves detection efficiency by introducing the Region Proposal Network (RPN).

[0007] YOLO series: The YOLO (You Only Look Once) series of algorithms is a one-stage target detection method that simplifies the target detection problem into a regression problem and directly predicts the object category and bounding box in the image. The advantage of YOLO lies in its real-time performance. Since it only requires one forward propagation to complete the entire detection process, it is much faster than the two-stage detection method. From version v1 to the current YOLOv10, the YOLO algorithm has continuously struck a balance between accuracy and speed, becoming the mainstream algorithm for real-time detection.

[0008] SSD (Single Shot MultiBox Detector): SSD is also a single-stage detection algorithm. It directly predicts targets on multi-scale feature maps and can effectively deal with object detection problems of different scales. SSD's multi-scale detection strategy gives it a great advantage in speed, but it does not perform as well as the two-stage detection method in small target detection.

[0009] Disadvantages of existing technology:

[0010] 1. The SSD algorithm has a slow response speed and a high false detection rate: The algorithm performs poorly for multi-scale target detection, especially for small targets that are critical to vehicle driving safety. It is prone to missed detection or false detection. At the same time, its real-time performance is poor and it cannot provide sufficiently timely and accurate detection results in complex driving environments.

[0011] 2. The YOLO algorithm has weak generalization ability in dealing with complex scenes: Although the YOLO algorithm performs well in a relatively ideal environment, it has weak generalization ability when facing the diversity and complexity of the driving environment and is difficult to adapt to the changes in different scenes. This limits its wide application in actual driving.

[0012] 3. Existing technologies are easily disturbed by irrelevant and erroneous information: In complex environments, the system is easily disturbed by erroneous and irrelevant information, which causes the model to misjudge during target detection and decision-making, affecting the accuracy of the detection results. These problems will reduce the user experience and thus affect driving safety. Summary of the invention

[0013] The technical problem to be solved by the present invention is to provide a significant target detection method for an intelligent assisted driving system in view of the deficiencies in the prior art.

[0014] The technical solution of the present invention is as follows:

[0015] A method for detecting a salient target in an intelligent assisted driving system comprises the following steps:

[0016] Step 1: Using the vehicle-mounted image acquisition device, the driving scene image based on the driver's visual area is acquired in real time and pre-processed;

[0017] Step 2: The backbone network performs underlying multi-scale feature extraction, interaction, and global enhancement of feature information to extract rich multi-scale feature information;

[0018] Step 3: The neck network includes an AESwinTransformer attention module, a cross-layer feature guided fusion module in the TFG network, and a multi-level feature interactive fusion module to fuse the multi-level features extracted from the backbone network into one piece;

[0019] Step 4: Detect and classify feature maps

[0020] The detection head in the detection head network is a dynamic detection head, which dynamically adjusts the detection strategy according to different detection scenarios based on the feature image transmitted from the neck network;

[0021] Step 5: Mix the adversarial samples and normal samples into the model, and use the trained model for adversarial training.

[0022] In the method described, in the step 2, after the preprocessing is completed, the image is passed into the network to start feature extraction in the backbone of the Y-AESTFG network. In this stage, the image is firstly extracted by the convolution module, which includes 3x3 convolution, standardization, and activation function ReLU; in addition, after obtaining the preliminary feature map, the feature map will be further extracted and processed by the interaction module. The interaction module is composed of the superposition of deep convolution and local multi-scale feature interaction modules, which can reduce the amount of model parameters while expanding the receptive field, so that the model can better extract rich multi-scale features in the image, such as Figure 1The interaction module shown is composed of the superposition of four groups of deep convolution and local multi-scale feature interaction modules in stage2-5; finally, the global background information enhancement module is used to capture features of various scales and the global context features of the entire image.

[0023] The method described in step 2 uses multiple convolution kernels of the same scale to replace convolution kernels of different scales in the local multi-scale feature interaction module to extract multi-scale local feature information in the image, thereby expanding the receptive field while keeping the model parameters reduced; then, feature interaction is used to make the final comprehensive feature have multi-scale information; and detail information is transferred between different layers through jump connections; assuming that the input feature map is F i After multiple 3x3 convolutions of the same scale, the output feature map F i LMIF The formula can be expressed as:

[0024]

[0025] Among them, F i represents the input feature map of the i-th layer, represents the weight matrix of the jth 3x3 convolution kernel, * represents the convolution operation, and n represents the number of superimposed 3x3 convolution layers. By continuously superimposing 3x3 convolution layers, it is equivalent to achieving a convolution effect with a larger receptive field, thereby achieving feature information extraction of different scales while keeping the number of model parameters small.

[0026] The method, in the step 2, after the local multi-scale feature interaction is performed after the last layer of convolution feature extraction, the obtained feature image is passed through the global background information enhancement module, and the input feature map is divided into multiple feature map subsets, so that different parts of the feature map can be processed independently, and the expansion DSConv3x3 with different expansion rates is applied to the feature map subsets obtained after segmentation to obtain new feature map subsets, and then cascade, jump connection and convolution operations are used to realize feature interaction and update feature map subsets; then the feature subsets are reintegrated to form new feature maps Element summation; also uses Conv1×1 operation and channel attention rearrangement Finally, the rearranged is fed into different convolutional blocks in multiple different guidance paths to generate personalized guidance information F i GCIE , extract the global background information of the entire image and fuse it with the local features obtained in stage5 to enhance the ability to perceive targets in complex backgrounds; the corresponding formula is as follows:

[0027]

[0028] Among them, G represents the global feature map, F l (i, j) represents the value of pixel (i, j) in the feature map of layer l; then it is fused with local features through the self-attention mechanism to obtain the enhanced feature map F l GCIE :

[0029] F l GCIE =Attention(F l ,G).

[0030] In the method described, in step three, the cross-layer feature guided fusion module, the input of each cross-layer feature guided module is composed of features extracted from two adjacent encoder layers in the backbone network. Since the resolutions of the two layers of feature maps are different, before performing the guidance operation, the low-resolution feature map needs to be upsampled first to ensure the consistency of the feature map resolution. Subsequently, the two layers of feature maps are fused by element-by-element multiplication, and the fused feature maps are feature guided using nested residual-like connections. This guidance method ensures that low-level and high-level features can converge to the same target at the same time. The cross-layer feature guided fusion process effectively promotes the coordination of features at different levels and the integration of information, guiding the low-level high-resolution features to fuse with the high-level low-resolution feature maps obtained from the backbone network. The formula is as follows:

[0031]

[0032] in, represents high-resolution features from low levels, F i high Represents low-resolution features from high-level layers, and α and β are learnable weight parameters. This module can improve the model's ability to detect small targets.

[0033] In the method described, in step 3, the feature map obtained after the cross-layer feature guided fusion module is continuously passed to the AESwinTransformer module, in which a stronger global information perception capability is introduced through the self-attention mechanism to better express the cross-level salient features in complex background information. The specific formula is as follows:

[0034]

[0035] Where Q, K, V represent query, key and value vectors respectively, and d is the scaling factor;

[0036] In the method, in step 3, in the multi-level feature interaction fusion module, the cross-level guidance features, the enhanced global guidance features, the local multi-scale features and the output of the previous AESwinTransformer module are used together to decode the salient targets in the driver's visual area in the driving scene. In the encoding process, the multiple features are first aligned to the same scale. Then, the global guidance information F i GCIE Embedded into feature F LMIF and feature F CGF (where the output F of the AESwinTransformer module Stage-1 The global guidance information is already included) to enhance the semantic and location information of traffic salient objects in the low-level features. Finally, multiple features are aggregated by element-wise summation and the Conv 1x1 operation is used to reduce the number of channels. The decoding process can be formulated as:

[0037]

[0038] To achieve effective fusion and decoding of multi-level features.

[0039] The method described in step 4 firstly obtains the feature map F∈R from the main network and the neck network H*W*C Passed to the detection head, where H, W, C represent the height, width and number of channels of the feature map respectively;

[0040] The detection effect is improved by weighting the importance of each channel, and the channel-level feature vector is generated using global average pooling (GAP):

[0041]

[0042] Then, two fully connected layers are used to generate the dynamic channel weight vector w c ∈R c :

[0043] w c =σ(W1ReLU(W2z+b2)+b1)

[0044] Among them, W1 and W2 are trainable weight matrices, b2 is the bias term of the first fully connected layer, which is usually used together with W2 to convert the global average pooling feature vector z into an intermediate feature vector, and b1 is the bias term of the second fully connected layer, which is used to weight the output of the intermediate feature vector in the previous step and generate the final channel attention weight w c ;

[0045] The spatial attention mechanism is used to generate a spatial attention map to highlight the salient objects in the image. Two spatial feature maps F are generated by using the maximum pooling and average pooling in the channel dimension. avg and F max :

[0046] F avg =AvgPool(F)

[0047] F max =MaxPool(F)

[0048] The two spatial feature maps are concatenated in the channel dimension and three 3x3 convolution operations are performed to generate the spatial attention map w. s ∈R H*W :

[0049] w s =σ(Conv3x3(Conv3x3(Conv3x3([F avg ,F max ]))))

[0050] After generating the dynamic channel attention weights and spatial attention maps, these weights and maps are then applied to the feature map F in the detection head to generate the dynamic feature map F':

[0051]

[0052] in, represents element-by-element multiplication. Through this operation, the channel and position information useful for salient target detection in the dynamic feature image based on the driver's gaze area is amplified;

[0053] The dynamic detection head detects the generated dynamic feature map, uses the generated dynamic weight parameters, inputs the dynamic feature map into the classification branch, reduces the input feature map F' to the number of categories k through dimensionality reduction convolution, generates a single category probability through global pooling, and finally generates a normalized category probability P through softmax activation to obtain the probability of the final significant target category; the relevant formula is as follows:

[0054] F cls =Conv1x1(F')

[0055] F cls pool = GAP(F cls )

[0056]

[0057] Where P i represents the probability of category i, and K is the number of categories.

[0058] The dynamic feature map F' is input to the regression branch. The feature map is first convolved with a 1x1 dimension reduction, and the output dimension is 4 to predict the parameters of the bounding box. Finally, the salient target is labeled by the decoding of the bounding box:

[0059]

[0060] Among them, x and y represent the coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box.

[0061] Use a multi-task loss function to optimize the predictions of the classification and regression branches, and use the cross entropy loss function on the classification loss to measure the predicted value P and the actual category label P * The difference between the two is as follows:

[0062]

[0063] The classic IOU intersection loss is used on the regression loss to measure the predicted bounding box P and the true bounding box P * The error is given by the following formula:

[0064] L reg =1-IoU(B,B * )

[0065] Combine the classification loss and regression loss into a total loss to optimize the entire model:

[0066] L=α·L cls +β·L reg

[0067] Among them, α and β are hyperparameters used to control the weights of classification loss and regression loss.

[0068] In the method, in step 5, the adversarial sample introduces a small disturbance into the image to simulate the interference in a complex environment. Assuming the batch size is N, each batch contains N / 2 normal samples x and their true labels y, and N / 2 adversarial samples x_adv and the same label y;

[0069] First, for each normal sample x, the loss L(x,y) is calculated through forward propagation. A new adversarial training loss function DRD-LOSS is defined. This loss function not only includes the loss of detecting the target position and category, but also includes the adversarial loss in adversarial training. According to the loss gradient, the FGSM generation method is used to generate adversarial perturbations and obtain adversarial samples x_adv. The formula is as follows:

[0070]

[0071] The perturbation range ε ​​should be set to a size suitable for the driving environment, ensuring that it does not affect the human eye's recognition of the target but can enhance the robustness of the model;

[0072] Input a mixed batch of normal samples and adversarial samples to the model and perform a complete forward propagation. Input batch X_mix = [x_1, x_2, ..., x_(N / 2), x_adv_1, x_adv_2, ..., x_adv_(N / 2)] passes through the model and obtains the predicted output y_pred_mix;

[0073] For each sample in the mixed input batch, calculate its corresponding detection loss. For normal samples x and adversarial samples x_adv, calculate the classification loss and bounding box regression loss respectively; for the losses L(x_i, y_i) and L(x_adv_i, y_i) of each sample i in the mixed batch, combine all losses and calculate the total loss L_total:

[0074] L_total=λ*(1 / N)*∑L(x,y)+(1-λ)*(1 / N)*∑L(x_adv,y)

[0075] Update the weight balance parameter λ: Optimize the model's weight λ through gradient descent to control the weights of normal sample and adversarial sample losses, so that it has stronger anti-interference ability when dealing with complex driving scenarios (such as low light, bad weather, occlusion, etc.). Based on the mixed loss L_total, back propagate and calculate the gradient of each layer. Use the optimizer SGD to update the model parameters to reduce the mixed loss and ensure that the model can adapt to the noise of adversarial samples while maintaining the detection accuracy of normal samples.

[0076] By adopting the above scheme, the present invention has the following beneficial effects:

[0077] 1. Improve the detection accuracy of small salient targets: By introducing AESwinTransformer to improve the accuracy of salient target detection, especially in the detection of salient small targets, it can effectively reduce missed detection and false detection. This method can enhance the system's target perception ability in complex environments.

[0078] 2. Improve the generalization and robustness of the model in complex environments: Improvements are made in the backbone network, neck network, and head detection module of YOLOv8 to further improve its detection performance. This allows the model to not only maintain the good real-time performance of YOLOv8, but also make full use of contextual background information to better extract features from complex background information. This will significantly improve the generalization and robustness of the model in complex driving environments.

[0079] 3. Reduce the interference of irrelevant scene information: By adding adversarial training to the model training process, the interference of irrelevant scene information on the intelligent assisted driving system can be greatly reduced. This method can effectively improve the prediction accuracy of the model, reduce misjudgment, and enhance the user experience and driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Network structure diagram of Y-AESTFG;

[0081] Figure 2 The structure diagram of the local multi-scale feature interaction module;

[0082] Figure 3 The structure diagram of the global background information enhancement module;

[0083] Figure 4 The structure diagram of the local feature guided fusion module;

[0084] Figure 5 Multi-level feature interaction fusion module structure diagram;

[0085] Figure 6 is the test result of Y-AESTFG; DETAILED DESCRIPTION

[0086] The present invention is described in detail below in conjunction with specific embodiments.

[0087] A method for detecting a salient target in an intelligent assisted driving system comprises the following steps:

[0088] Step 1: Use the vehicle-mounted image acquisition device to obtain the driving scene image based on the driver's visual area in real time and perform preprocessing. The specific steps are as follows:

[0089] In the step 1, image normalization is first performed, that is, the pixel values ​​of the input image are scaled to the range of [0, 1] to reduce the impact of brightness differences;

[0090] In the step 1, the resolution of the normalized input image is adjusted to 640×640 to ensure the consistency of the model input;

[0091] In step 1, after the resolution is adjusted and unified, data augmentation, rotation, translation, cropping, noise addition and other operations are performed to expand the diversity of the data set and enhance the generalization ability of the model. This step helps the model maintain stable performance in different driving scenarios (such as occlusion, low light, weather changes, etc.).

[0092] Step 2: Build a pytorch virtual environment and configure the model training environment;

[0093] In step 2, some specific training environments are configured: python = 3.11 pytorch = 2.1.1, cuda = 11.8;

[0094] Step 3: The backbone network performs underlying multi-scale feature extraction, interaction, and global enhancement of feature information to extract rich multi-scale feature information;

[0095] In step 3, after the preprocessing is completed, the image is passed into the network and the feature extraction is started in the backbone of the Y-AESTFG network, such as Figure 1 As shown in the figure, at this stage, the convolution module is first used to perform preliminary feature extraction on the image. The convolution module includes 3x3 convolution, standardization, and activation function ReLU. In addition, after obtaining the preliminary feature map, the interaction module will continue to extract and process the local multi-scale features of the feature map. The interaction module is composed of the superposition of deep convolution and local multi-scale feature interaction modules, which can reduce the number of model parameters while expanding the receptive field, so that the model can better extract rich multi-scale features in the image, such as Figure 1 The interaction module shown is composed of the superposition of four groups of deep convolution and local multi-scale feature interaction modules in stage2-5; finally, the global background information enhancement module is used to capture features of various scales and the global context features of the entire image.

[0096] In step 3, multiple convolution kernels of the same scale are used in the local multi-scale feature interaction module (such as Figure 2 In this way, we use a 3x3 kernel and two 3x3 kernels to replace convolution kernels of different scales to extract multi-scale local feature information in the image. This can expand the receptive field while keeping the number of model parameters small. Then, we use feature interaction to make the final comprehensive feature have multi-scale information. And we use jump connections to transfer detail information between different layers. Assume that the input feature map is F i After multiple 3x3 convolutions of the same scale, the output feature map F i LMIF The formula can be expressed as:

[0097]

[0098] Among them, F i represents the input feature map of the i-th layer, represents the weight matrix of the jth 3x3 convolution kernel, * represents the convolution operation, and n represents the number of superimposed 3x3 convolution layers. By continuously superimposing 3x3 convolution layers, it is equivalent to achieving a convolution effect with a larger receptive field. For example, the equivalent receptive field of two superimposed 3x3 convolution layers is 5x5, and the equivalent receptive field of three superimposed 3x3 convolution layers is 7x7, thereby achieving feature information extraction of different scales while keeping the number of model parameters small.

[0099] In the step 3, if Figure 1 As shown, after the local multi-scale feature interaction after the last layer (stage 5) of convolution feature extraction, the obtained feature image is passed through the global background information enhancement module 11 (specifically as shown in Figure 3 ), the input feature map is divided into multiple feature map subsets in a certain way (such as Figure 3 After segmentation, ), so that different parts of the feature map can be processed independently, and the expansion DSConv3x3 with different expansion rates is applied to the feature map subsets obtained after segmentation to obtain new feature map subsets ( ), then, we use cascade, skip connection and convolution operations to realize feature interaction and update feature map subsets. Afterwards, we reintegrate the feature subsets to form new feature maps Element summation. It also uses Conv1×1 operation and channel attention rearrangement Finally, the rearranged is fed into different convolutional blocks in multiple different guidance paths to generate personalized guidance information F i GCIE , extract the global background information of the entire image, and Figure 1 The local features obtained in stage 5 are fused to enhance the ability to perceive targets in complex backgrounds. The corresponding formula is as follows:

[0100]

[0101] Among them, G represents the global feature map, F l (i,j) represents the feature map of layer l (corresponding to Figure 1 The value of the pixel (i, j) in stage2, 3, 4, 5). Then the enhanced feature map F is obtained by fusing it with the local features through the self-attention mechanism. l GCIE :

[0102] F l GCIE =Attention(F l ,G)

[0103] Step 4: The neck network includes the AESwinTransformer attention module (AdvancedEfficientSwinTransformer), the cross-layer feature guidance fusion module in the TFG (Traffic Feature Deep Interaction and GuidanceFusion) network, and the multi-level feature interaction fusion module, which fuses the multi-level features extracted from the backbone network into one piece.

[0104] In step 4, the cross-layer feature guided fusion module (specifically, Figure 4 ), the input of each cross-layer feature guidance module consists of the features extracted from two adjacent encoder layers in the backbone network. Since the resolutions of the two layers of feature maps are different, the low-resolution feature maps need to be upsampled before the guidance operation is performed to ensure the consistency of the feature map resolution. Subsequently, the two layers of feature maps are fused by element-wise multiplication, and the fused feature maps are feature guided using nested residual-like connections. This guidance method ensures that low-level and high-level features can converge to the same target at the same time. The cross-layer feature guidance fusion process effectively promotes the coordination of features at different levels and the integration of information, guiding the low-level high-resolution features to fuse with the high-level low-resolution feature maps obtained from the backbone network. The formula is as follows:

[0105]

[0106] in, represents high-resolution features from low levels, F i high Represents low-resolution features from high-level layers, and α and β are learnable weight parameters. This module can improve the model's ability to detect small targets.

[0107] In step 4, the feature map obtained after the cross-layer feature guided fusion module is continuously passed to the AESwinTransformer module, in which a stronger global information perception capability is introduced through the self-attention mechanism to better express the cross-level salient features in the complex background information. The specific formula is as follows:

[0108]

[0109] Where Q, K, V represent query, key and value vectors respectively, and d is the scaling factor;

[0110] In the step 4, in the multi-level feature interactive fusion module (such as Figure 5As shown in Figure 2, the cross-level guidance features, enhanced global guidance features, local multi-scale features, and the output of the previous AESwinTransformer module are used together to decode the salient targets in the driving scene based on the driver's visual area. During the encoding process, multiple features are first aligned to the same scale. Then, the global guidance information F i GCIE Embedded into feature F LMIF and feature F CGF (where the output F of the AESwinTransformer module Stage-1 The global guidance information is already included) to enhance the semantic and location information of traffic salient objects in the low-level features. Finally, multiple features are aggregated by element-wise summation and the Conv 1x1 operation is used to reduce the number of channels. The decoding process can be formulated as:

[0111]

[0112] To achieve effective fusion and decoding of multi-level features.

[0113] Step 5: Detect and classify feature maps

[0114] In step 5, in order to solve the problem that yolov8 does not work well in processing dynamically changing salient targets, the detection head in the detection head network of this model is a dynamic detection head. First, the feature map F∈R H*W*C Passed to the detection head, where H, W, C represent the height, width and number of channels of the feature map respectively;

[0115] In step 5, the key lies in the generation of the channel attention mechanism. The multi-scale feature map output from the neck network is used as input to generate the dynamic channel attention weight of the detection head, and the detection effect is improved by weighting the importance of each channel. Global average pooling (GAP) is usually used to generate channel-level feature vectors:

[0116]

[0117] Then, two fully connected layers are used to generate the dynamic channel weight vector w c ∈R c :

[0118] w c =σ(W1ReLU(W2z+b2)+b1)

[0119] Among them, W1 and W2 are trainable weight matrices, b2 is the bias term of the first fully connected layer, which is usually used together with W2 to convert the global average pooling feature vector z into an intermediate feature vector, and b1 is the bias term of the second fully connected layer, which is used to weight the output of the intermediate feature vector in the previous step and generate the final channel attention weight w c .

[0120] In step 5, a spatial attention map is generated using a spatial attention mechanism to highlight the salient objects in the image. Two spatial feature maps F are generated by using the maximum pooling and average pooling in the channel dimension. avg and F max :

[0121] F avg =AvgPool(F)

[0122] F max =MaxPool(F)

[0123] The two spatial feature maps are concatenated in the channel dimension and three 3x3 convolution operations are performed to generate the spatial attention map w. s ∈R H*W :

[0124] w s =σ(Conv3x3(Conv3x3(Conv3x3([F avg ,F max ]))))

[0125] In step 5, after generating the dynamic channel attention weights and spatial attention maps, these weights and maps are then applied to the feature map F in the detection head to generate a dynamic feature map F':

[0126]

[0127] in, Represents element-by-element multiplication. Through this operation, the channel and position information useful for salient target detection in the dynamic feature image based on the driver's gaze area is amplified.

[0128] In step 5, the dynamic detection head detects the generated dynamic feature map, uses the generated dynamic weight parameters, inputs the dynamic feature map into the classification branch, reduces the input feature map F' to the number of categories k through dimensionality reduction convolution, generates a single category probability through global pooling, and finally generates a normalized category probability P through softmax activation to obtain the probability of the final significant target category. The relevant formula is as follows:

[0129] F cls =Conv1x1(F')

[0130] F cls pool =GAP(F cls )

[0131]

[0132] Where P i represents the probability of category i, and K is the number of categories.

[0133] In step 5, the dynamic feature map F' is input into the regression branch, and a 1x1 dimensionality reduction convolution is performed on the feature map, and the output dimension is 4, which is used to predict the parameters of the bounding box. Finally, the salient objects are labeled by the decoding of the bounding box:

[0134]

[0135] Among them, x and y represent the coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box.

[0136] In step 5, a multi-task loss function is usually used to optimize the prediction of the classification and regression branches. The present invention uses a cross entropy loss function on the classification loss to measure the predicted value P and the actual class label P * The difference between the two is as follows:

[0137]

[0138] The classic IOU intersection loss is used on the regression loss to measure the predicted bounding box P and the true bounding box P * The error is given by the following formula:

[0139] L reg =1-IoU(B,B * )

[0140] Combine the classification loss and regression loss into a total loss to optimize the entire model:

[0141] L=α·L cls +β·L reg

[0142] Among them, α and β are hyperparameters used to control the weights of classification loss and regression loss.

[0143] Step 6: Mix the adversarial samples and normal samples into the model, and use the trained model for adversarial training.

[0144] In step 6, adversarial samples are generated. During the training process, the adversarial sample generator is used to generate samples with disturbances. These adversarial samples introduce small disturbances in the image to simulate interference in complex environments (such as noise, occlusion, etc.). Assuming the batch size is N, each batch contains N / 2 normal samples x and their true labels y, and N / 2 adversarial samples x_adv and the same label y.

[0145] In step 6, first, for each normal sample x, the loss L(x, y) is calculated by forward propagation. A new adversarial training loss function DRD-LOSS is defined. This loss function not only includes the loss of detecting the target position and category, but also includes the adversarial loss in adversarial training. According to the loss gradient, the FGSM generation method is used to generate adversarial perturbations to obtain adversarial samples x_adv. The formula is as follows:

[0146]

[0147] The disturbance range ε ​​should be set to a size suitable for the driving environment to ensure that it does not affect the target recognition of the human eye but can enhance the robustness of the model.

[0148] In step 6, a mixed batch of normal samples and adversarial samples is input into the model for a complete forward propagation. The input batch X_mix = [x_1, x_2, ..., x_(N / 2), x_adv_1, x_adv_2, ..., x_adv_(N / 2)] is passed through the model to obtain the predicted output y_pred_mix.

[0149] In step 6, for each sample in the mixed input batch, the corresponding detection loss is calculated. For the normal sample x and the adversarial sample x_adv, the classification loss and the bounding box regression loss are calculated respectively. For the losses L(x_i, y_i) and L(x_adv_i, y_i) of each sample i in the mixed batch, all losses are combined to calculate the total loss L_total:

[0150] L_total=λ*(1 / N)*∑L(x,y)+(1-λ)*(1 / N)*∑L(x_adv,y)

[0151] Update the weight balance parameter λ: Optimize the model's weight λ through gradient descent to control the weights of normal sample and adversarial sample losses, so that it has stronger anti-interference ability when dealing with complex driving scenarios (such as low light, bad weather, occlusion, etc.). Based on the mixed loss L_total, back propagate and calculate the gradient of each layer. Use the optimizer SGD to update the model parameters to reduce the mixed loss and ensure that the model can adapt to the noise of adversarial samples while maintaining the detection accuracy of normal samples.

[0152] Following the above steps, the process of generating adversarial examples, mixing inputs, and calculating losses is repeated in each training batch, ensuring that the model continuously learns the characteristics of normal and adversarial examples throughout the training process.

[0153] Experimental results and analysis

[0154] Table 1 Experimental environment

[0155]

[0156] 2 Datasets

[0157] BDD100K dataset: The BDD100K (Berkeley DeepDrive 100K) dataset is a large-scale autonomous driving dataset that contains image data under diverse road conditions, weather, time, geographic location, and different environmental conditions. This dataset is mainly used to study tasks such as target detection, image segmentation, tracking, and behavior prediction in autonomous driving. It contains 100,000 driving video image frames and labels, and is one of the largest and most diverse autonomous driving datasets. This article selects 17,333 images, including a training set (15,333 images), a validation set (1,000 images), and a test set (1,000 images).

[0158] 3. Comparative test and ablation experiment:

[0159] The following are hypothetical experimental results based on the BDD100k dataset. The evaluation indicators include accuracy, recall, parameters, and mean average precision (mAP). Through ablation experiments, different modules are gradually removed or replaced to analyze the impact of each module on the overall performance of the model. The results are shown in Table 2.

[0160] The Y-AESTFG model is compared with the original YOLOv8, YOLOv5, SSD and other models, and the performance of the model is evaluated using indicators such as mAP, precision, recall, and detection time.

[0161] Table 2 Ablation experiment data on BDD100K dataset

[0162]

[0163] Table 3 Comparison with mainstream algorithms on the BDD100K dataset

[0164] Method P / ％ R / ％ mAP@0.5% Inference Time(ms) YOLOv5s 80.1 76.5 75.4 14.8 SSD 75 70.3 69.8 13.1 YOLOv8n 82.3 77.9 78.5 15.2 Y-AESTFG (non-adversarial training) 87 76.3 85.2 17.6 Y-AESTFG (with adversarial training) 90.3 86.2 85.8 18.3

[0165] From the results in Table 3

[0166] 1. Mean Average Precision (mAP): The improved YOLOv8 model significantly improves mAP, especially after the introduction of adversarial training, reaching 85.8%, 7.3% higher than the original YOLOv8. The local multi-scale feature interaction module and the global background information enhancement module jointly improve the model's ability to capture salient targets.

[0167] 2. Precision and Recall: The precision and recall of the improved YOLOv8 are improved by 8% and 8.3% respectively compared with the original model. This shows that adversarial training can significantly reduce missed detections and false detections, making the model more reliably detect objects of different sizes and backgrounds.

[0168] 3. Robustness: In the complex scene test set (such as rainy days, nighttime, and occlusion), the improved YOLOv8 with adversarial training showed better robustness and significantly reduced false positive rate. This is mainly due to the adversarial loss added to the loss function by adversarial training, which enables the model to better resist interference from different scenes and adversarial perturbations.

[0169] 4. Real-time performance (Inference Time): Although the inference time of the improved YOLOv8 is slightly longer than that of the original YOLOv8 (about 3.1ms), it is still able to meet the needs of real-time detection within the requirements of the intelligent driving system. This slight performance overhead is acceptable, especially when the detection accuracy is significantly improved.

[0170] 5 Comparison of salient object detection results

[0171] To demonstrate the detection capability of Y-AESTFG, visualization results are provided. Figure 6 (a) The original image, (b) the detection results of YOLOv8n, and (c) the detection results of Y-AESTFG are shown. Compared with YOLOv8n, Y-AESTFG shows higher precision and recall in salient object detection, and shows good performance in detecting some small salient objects. In addition, the model proposed in this paper improves the detection speed and the anti-interference ability is significantly improved, which is better than YOLOv8n, which is very beneficial for the real-world application of salient object detection in intelligent assisted driving systems.

[0172] The present invention introduces local multi-scale feature interaction and global information enhancement modules into the YOLOv8 backbone network, adds AESwinTransformer, cross-layer feature guided fusion, and multi-level feature interaction fusion modules into the neck network, and improves the detection head to a dynamic detection head, combined with the adversarial training mechanism, effectively improving the model's significant target detection capability in the intelligent assisted driving system. This solution not only improves the accuracy and robustness of detection, but also enhances the adaptability to complex driving scenes, and is suitable for various complex driving environments.

[0173] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for detecting a salient target in an intelligent assisted driving system, characterized in that: The following steps are involved: Step 1: Using the vehicle-mounted image acquisition device, the driving scene image based on the driver's visual area is acquired in real time and pre-processed; Step 2: The backbone network performs underlying multi-scale feature extraction, interaction, and global enhancement of feature information to extract rich multi-scale feature information; Step 3: The neck network includes the AESwinTransformer attention module, the cross-layer feature guided fusion module in the TFG network, and the multi-level feature interactive fusion module, which fuses the multi-level features extracted from the backbone network into one piece; Step 4: Detect and classify feature maps The detection head in the detection head network is a dynamic detection head, which dynamically adjusts the detection strategy according to different detection scenarios based on the feature image transmitted from the neck network; Step 5: Mix the adversarial samples and normal samples into the model, and use the trained model for adversarial training.

2. The method according to claim 1, characterized in that In the step 2, after the preprocessing is completed, the image is passed into the network to start feature extraction in the backbone of the Y-AESTFG network. In this stage, the image is firstly subjected to preliminary feature extraction through the convolution module, which includes 3x3 convolution, standardization, and activation function ReLU. In addition, after obtaining the preliminary feature map, the feature map will be further subjected to local multi-scale feature extraction and processing through the interaction module. The interaction module is composed of the superposition of deep convolution and local multi-scale feature interaction modules, which can expand the receptive field while reducing the number of model parameters, so that the model can better extract rich multi-scale features in the image. Finally, the global background information enhancement module is used to capture features of various scales and the global context features of the entire image.

3. The method according to claim 1, characterized in that In the step 2, multiple convolution kernels of the same scale are used in the local multi-scale feature interaction module to replace convolution kernels of different scales to extract multi-scale local feature information in the image, so as to expand the receptive field while keeping the number of model parameters reduced; then feature interaction is used to make the final comprehensive feature have multi-scale information; and detail information is transferred between different layers through jump connections; assuming that the input feature map is F i After multiple 3x3 convolutions of the same scale, the output feature map is The formula can be expressed as: Among them, F i represents the input feature map of the i-th layer, represents the weight matrix of the jth 3x3 convolution kernel, * represents the convolution operation, and n represents the number of superimposed 3x3 convolution layers. By continuously superimposing 3x3 convolution layers, a convolution effect with a larger receptive field is achieved, thereby extracting feature information of different scales while keeping the number of model parameters small.

4. The method according to claim 1, characterized in that: In the step 2, after the local multi-scale feature interaction is performed after the last layer of convolutional feature extraction, the obtained feature image is passed through the global background information enhancement module, and the input feature map is divided into multiple feature map subsets, so that different parts of the feature map can be processed independently, and the expansion DSConv3x3 with different expansion rates is applied to the feature map subsets obtained after segmentation to obtain new feature map subsets, and then cascade, jump connection and convolution operations are used to realize feature interaction and update feature map subsets; then the feature subsets are reintegrated to form a new feature map Element summation; also uses Conv1×1 operation and channel attention rearrangement Finally, the rearranged Feed into different convolutional blocks in multiple different guidance paths to generate personalized guidance information Extract the global background information of the entire image and fuse it with the local features obtained in stage 5 to enhance the ability to perceive targets in complex backgrounds; the corresponding formula is as follows: Among them, G represents the global feature map, F l (i, j) represents the value of pixel (i, j) in the feature map of layer l; then it is fused with local features through the self-attention mechanism to obtain the enhanced feature map 5. The method according to claim 1, characterized in that In the step 3, the input of each cross-layer feature guidance module is composed of features extracted from two adjacent encoder layers in the backbone network; since the resolutions of the two layers of feature maps are different, before performing the guidance operation, the low-resolution feature map needs to be upsampled first to ensure the consistency of the feature map resolution; then, the two layers of feature maps are fused by element-wise multiplication, and the fused feature maps are feature guided using nested residual-like connections; this guidance method ensures that low-level and high-level features can converge to the same target at the same time; the cross-layer feature guidance fusion process effectively promotes the coordination of features at different levels and the integration of information, guiding the low-level high-resolution features to fuse with the high-level low-resolution feature maps obtained from the backbone network, and the formula is as follows: in, represents high-resolution features from low levels, Represents low-resolution features from high levels, α and β are learnable weight parameters; this module can improve the model's detection capability for small targets.

6. The method according to claim 1, characterized in that In step 3, the feature map obtained after the cross-layer feature guided fusion module is continuously passed to the AESwinTransformer module, in which a stronger global information perception capability is introduced through the self-attention mechanism to better express the cross-level salient features in complex background information. The specific formula is as follows: Where Q, K, V represent query, key and value vectors respectively, and d is the scaling factor.

7. The method according to claim 1, characterized in that In the step 3, in the multi-level feature interactive fusion module, the cross-level guidance features, the enhanced global guidance features, the local multi-scale features and the output of the previous AESwinTransformer module are jointly used to decode the salient targets in the driver's visual area during the driving scene; in the encoding process, the multiple features are first aligned to the same scale; then, the global guidance information is combined into a Embedded into feature F LMIF and feature F CGF In order to enhance the semantic and location information of traffic salient targets in low-level features; finally, multiple features are aggregated by element-wise summation and Conv 1x1 operation is used to reduce the number of channels; the decoding process can be formulated as: To achieve effective fusion and decoding of multi-level features.

8. The method according to claim 1, characterized in that In the step 4, firstly, the feature map F∈R from the trunk network and the neck network is obtained. H*W*C Passed to the detection head, where H, W, C represent the height, width and number of channels of the feature map respectively; The detection effect is improved by weighting the importance of each channel, and the channel-level feature vector is generated using global average pooling (GAP): Then, two fully connected layers are used to generate the dynamic channel weight vector w c ∈R c : In c =σ(W1ReLU(W2z+b2)+b1) Among them, W1 and W2 are trainable weight matrices, b2 is the bias term of the first fully connected layer, which is usually used together with W2 to convert the global average pooling feature vector z into an intermediate feature vector, and b1 is the bias term of the second fully connected layer, which is used to weight the output of the intermediate feature vector in the previous step and generate the final channel attention weight w c ; The spatial attention mechanism is used to generate a spatial attention map to highlight the salient objects in the image. Two spatial feature maps F are generated by using the maximum pooling and average pooling in the channel dimension. avg and F max : F avg =AvgPool(F) F max =MaxPool(F) The two spatial feature maps are concatenated in the channel dimension and three 3x3 convolution operations are performed to generate the spatial attention map w. s ∈R H*W : w s =σ(Conv3x3(Conv3x3(Conv3x3([F avg ,F max ])))) After generating the dynamic channel attention weights and spatial attention maps, these weights and maps are then applied to the feature map F in the detection head to generate the dynamic feature map F': in, represents element-by-element multiplication. Through this operation, the channel and position information useful for salient target detection in the dynamic feature image based on the driver's gaze area is amplified; The dynamic detection head detects the generated dynamic feature map, uses the generated dynamic weight parameters, inputs the dynamic feature map into the classification branch, reduces the input feature map F' to the number of categories k through dimensionality reduction convolution, generates a single category probability through global pooling, and finally generates a normalized category probability P through softmax activation to obtain the probability of the final significant target category; the relevant formula is as follows: F cls =Conv1x1(F') F cls pool =GAP(F cls ) Where P i represents the probability of category i, K is the number of categories; The dynamic feature map F' is input to the regression branch. The feature map is first convolved with a 1x1 dimension reduction, and the output dimension is 4 to predict the parameters of the bounding box. Finally, the salient target is labeled by the decoding of the bounding box: Among them, x and y represent the coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box; Use a multi-task loss function to optimize the predictions of the classification and regression branches, and use the cross entropy loss function on the classification loss to measure the predicted value P and the actual category label P * The difference between the two is as follows: The classic IOU intersection loss is used on the regression loss to measure the predicted bounding box P and the true bounding box P * The error is given by the following formula: L reg =1-IoU(B,B * ) Combine the classification loss and regression loss into a total loss to optimize the entire model: L=α·L cls +β·L reg Among them, α and β are hyperparameters used to control the weights of classification loss and regression loss.

9. The method according to claim 1, characterized in that: In step 5, the adversarial sample introduces a small perturbation into the image to simulate the interference in a complex environment. Assuming the batch size is N, each batch contains N / 2 normal samples x and their true labels y, and N / 2 adversarial samples x_adv and the same label y; First, for each normal sample x, the loss L(x, y) is calculated by forward propagation; a new adversarial training loss function DRD-LOSS is defined, which includes not only the loss of detecting the target position and category, but also the adversarial loss in adversarial training. According to the loss gradient, the FGSM generation method is used to generate adversarial perturbations and obtain adversarial samples x_adv. The formula is as follows: x_adv=x+ε*sign(▽_xL(x,y)) The perturbation range ε ​​should be set to a size suitable for the driving environment, ensuring that it does not affect the human eye's recognition of the target but can enhance the robustness of the model; Input a mixed batch of normal samples and adversarial samples into the model and perform a complete forward propagation; the input batch X_mix = [x_1, x_2, ..., x_(N / 2), x_adv_1, x_adv_2, ..., x_adv_(N / 2)] passes through the model and obtains the predicted output y_pred_mix; For each sample in the mixed input batch, calculate its corresponding detection loss; for normal samples x and adversarial samples x_adv, calculate the classification loss and bounding box regression loss respectively; for the losses L(x_i, y_i) and L(x_adv_i, y_i) of each sample i in the mixed batch, combine all losses and calculate the total loss L_total: L_total=λ*(1 / N)*∑L(x,y)+(1-λ)*(1 / N)*∑L(x_adv,y) Update the weight balance parameter λ: Optimize the model's weight λ through gradient descent to control the weights of normal sample and adversarial sample losses, so that it has stronger anti-interference ability when dealing with complex driving scenarios; perform back propagation based on the mixed loss L_total to calculate the gradient of each layer; use the optimizer SGD to update the model parameters to reduce the mixed loss and ensure that the model can adapt to the noise of adversarial samples while maintaining the detection accuracy of normal samples.

Images