Intelligent networked automobile multi-mode sensing and decision-making system based on deep learning

Abstract

The invention relates to the technical field of intelligent driving, and discloses an intelligent networked automobile multi-modal sensing and decision-making system based on deep learning, and the system comprises a multi-modal data collection module which is used for collecting image data and radar time sequence data around a vehicle through a vehicle-mounted camera and a vehicle-mounted radar sensor, the surrounding environment of the vehicle can be sensed in all directions; the image data processing module is used for processing the image data by using an improved deep learning model and extracting target object feature information in the image; the radar data processing module is used for analyzing the change trend of the radar time sequence data and making an optimal driving decision according to different traffic scenes in combination with the feature information of the target object; and the optimal decision execution module is used for executing an optimal decision action according to the optimal driving decision. According to the invention, high-precision perception and quick response to environment information in a complex traffic scene can be realized, the driving safety is improved, and the traffic efficiency is optimized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. Background Art

[0002] With the rapid development of the global automotive industry, intelligent connected vehicles have become the core research direction of future transportation. Traditional automotive perception and decision-making systems mainly rely on a single type of sensor, which has limited information range and accuracy, and cannot meet the needs of comprehensive perception of the vehicle's surroundings in complex traffic environments. In addition, such systems often have problems such as slow response speed and low decision-making accuracy when facing complex scenarios such as bad weather and traffic congestion, which has an adverse impact on driving safety and traffic efficiency.

[0003] In recent years, deep learning technology has made significant progress in image processing, target detection, and path planning. However, the current multimodal perception system still faces the following challenges: First, the accuracy of multi-sensor data fusion needs to be improved, especially in low signal-to-noise ratio or dynamic environments; second, the decision model is insufficient in terms of real-time performance and resource efficiency, and it is difficult to meet the requirements of intelligent connected vehicles for high-efficiency real-time processing.

[0004] Therefore, there is an urgent need to develop an intelligent connected vehicle perception and decision-making system that can efficiently integrate multimodal sensor data and has high-precision perception and rapid response capabilities, which is of great significance for improving driving safety and optimizing traffic efficiency. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention aims to propose a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. By designing an efficient data fusion algorithm and a real-time optimized decision-making model, the system can achieve high-precision perception and rapid response to environmental information in complex traffic scenarios, so as to improve driving safety and optimize traffic efficiency.

[0006] The purpose of the present invention can be achieved by the following technical solutions: In a first aspect, the present invention provides a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning, comprising: The multimodal data acquisition module is used to collect image data and radar time series data around the vehicle through the vehicle-mounted camera and vehicle-mounted radar sensor to achieve all-round perception of the vehicle's surrounding environment; An image data processing module, used to process the image data using an improved deep learning model to extract feature information of a target object in the image; wherein the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein the network model architecture is composed of a backbone network, a Transformer encoder, a category counting module, a counting-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder; A radar data processing module, used to analyze the changing trend of the radar time series data, and make optimal driving decisions according to different traffic scenarios in combination with the characteristic information of the target object; The optimal decision execution module is used to execute the optimal decision action according to the optimal driving decision.

[0007] Preferably, the step of processing the image data using an improved deep learning model to extract feature information of the target object in the image includes: The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels; Sending the multi-scale feature map to the Transformer encoder for self-attention operation to capture long-distance dependencies in the features and generate visual features enhanced by self-attention; The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances; The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map; The dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries; The efficient hybrid decoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent object recognition and bounding box regression.

[0008] Preferably, the category counting module is composed of a density extractor and a classification head; the density extractor is used to perform dilated convolution on the high-resolution feature map to generate a density map, and the classification head is used to classify the number of image instances into four levels according to the density map.

[0009] Preferably, the count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross attention and channel attention operations to obtain a count-guided enhanced feature map, including: The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations; Multiplying the spatial attention map with the original visual feature to obtain a spatially enhanced feature; A channel attention operation is performed on the spatially enhanced features to obtain a count-guided enhanced feature map.

[0010] Preferably, the dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries according to the classification result of the category counting module, and optimizes the query content and location, including: Determine the number of decoder queries K according to the classification result of the category counting module; use the counting to guide the enhanced feature map , based on the classification score, the top K features are selected, and the query content and position deviation are generated through linear transformation and FFN prediction respectively, and then the query position is adjusted to adapt it to the image target density to facilitate small target positioning.

[0011] The efficient hybrid decoder is a hybrid encoder based on attention-based intra-scale feature interaction and convolutional neural network-based cross-scale feature fusion; The efficient hybrid decoder performs in-scale interaction on high-level features output by the backbone network based on in-scale feature interaction of attention, and is used to perform self-attention operation only on high-level features to capture the relationship between conceptual entities in high-level features, while reducing the risk of repeated or invalid interaction on low-level features; Cross-scale feature fusion fuses adjacent scale features through fusion blocks, integrates adjacent scale features into a richer feature representation, and outputs the fused multi-scale features for subsequent target recognition and bounding box regression.

[0012] Preferably, the constructed loss function is a weighted combination of multiple losses, specifically including L1 loss , bounding box regression loss , category loss and classification loss Weighted loss function To optimize the detection task; the loss function for: ; in, , , , They correspond to L1 losses respectively. , bounding box regression loss , category loss and classification loss The weight coefficient of .

[0013] Preferably, the radar data processing module is used to analyze the change trend of the radar time series data and make the optimal driving decision according to different traffic scenarios in combination with the target object feature information, including: Analyze the change trend of the radar time series data to obtain the current state of the vehicle; Combining the target object feature information extracted by the image data processing module with the current state of the vehicle to form an enhanced state; Based on the enhanced state, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem; Build reward functions to encourage the model to make safe and efficient decisions; Build a decision-making support mechanism to trigger preset safety strategies in emergency situations; Under the reward function and the decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

[0014] Preferably, the decision optimization problem constructed by the radar data processing module is expressed as selecting the optimal control action under the current enhanced state to maximize the cumulative expected reward; The decision optimization problem is expressed as: ; in, : In the current enhanced state Execute an action The cumulative expected rewards received; : The optimal decision at the current time step t, i.e., the control command of the vehicle; : A set of optional actions, including acceleration, deceleration, and steering; Optimize the cumulative expected reward by using the policy gradient method , in order to select the optimal control action.

[0015] Preferably, the reward function is designed to encourage safe, efficient and comfortable driving behavior, including safety rewards, efficiency rewards, comfort rewards and object rewards; The decision support mechanism sets a series of emergency triggering conditions and corresponding control strategies.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) By integrating data from multiple sensors such as on-board cameras, millimeter-wave radars, and lidars, the system can perceive complex traffic environments more comprehensively and accurately. The improved deep learning model improves the recognition and positioning accuracy of target objects through customized category counting modules and counting-guided feature enhancement modules, especially in the case of small targets and occluded targets.

[0017] (2) The real-time optimized decision model uses an efficient hybrid decoder and a dynamic query selection module, which significantly improves decision-making speed and resource efficiency. By optimizing the cumulative expected reward through the policy gradient method, the system can quickly make the optimal driving decision in complex traffic scenarios to ensure driving safety.

[0018] (3) The system supports the integration and configuration of multiple sensors, and can flexibly adjust the type and quantity of sensors according to actual needs. By adjusting the width and depth of the efficient hybrid encoder and the parameters of the decoder, the system can achieve flexible model scaling and be suitable for different scenarios and vehicle types.

[0019] (4) Design a decision-making mechanism based on reward functions and rule assistance to encourage the model to make safe and efficient driving behaviors. In emergency situations, the system can trigger preset safety strategies, such as emergency braking or parking, to effectively reduce the risk of traffic accidents. By rationally planning routes and reducing unnecessary stops and waits, the system can significantly improve traffic efficiency and alleviate traffic congestion.

[0020] In summary, the multimodal perception and decision-making system for intelligent connected vehicles based on deep learning proposed in the present invention shows significant advantages in improving perception accuracy and robustness, improving decision-making speed and efficiency, enhancing system adaptability and scalability, reducing the risk of traffic accidents, and improving traffic efficiency. It is of great significance to promote the development and application of intelligent connected vehicle technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 It is a unit module diagram of a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided by an embodiment of the present invention; Figure 2 It is a flow chart of a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0024] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0025] The present invention proposes a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. By designing an efficient data fusion algorithm and a real-time optimized decision model, the system can achieve high-precision perception and rapid response to environmental information in complex traffic scenarios. Specific goals include: enhancing the system's perception of target objects and traffic environments through multimodal sensor data fusion; using an improved deep learning model, proposing a custom category counting module, a counting-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder to improve the computational efficiency and reliability of the decision-making algorithm; designing a decision-making mechanism based on reward functions and rule assistance to reduce the risk of traffic accidents and ensure that vehicles can perform operations quickly and accurately in a dynamic environment.

[0026] See also Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multimodal perception and decision-making system 100 for intelligent connected vehicles based on deep learning, comprising: The multimodal data acquisition module 110 is used to collect image data and radar time series data around the vehicle through the vehicle-mounted camera and the vehicle-mounted radar sensor to perform all-round perception of the vehicle's surrounding environment; Among them, multiple sensors such as vehicle-mounted cameras and vehicle-mounted radar sensors (such as millimeter-wave radars, laser radars, etc.) are used to perceive the vehicle's surrounding environment in all directions. The vehicle-mounted cameras, millimeter-wave radars, and laser radars are selected, installed, and debugged to ensure that they can obtain data stably and accurately.

[0027] Sensor selection and configuration: Car Camera: Selection: Choose a high-resolution, wide-angle camera to ensure that it can capture a wide field of view around the vehicle. At the same time, consider the camera's night vision function and anti-shake performance to improve the quality of data collection in bad weather and night driving.

[0028] Configuration: Install multiple cameras according to different positions of the vehicle (such as front, rear, and side) to achieve all-round monitoring of the vehicle's surroundings.

[0029] Millimeter wave radar: Selection: Choose a millimeter-wave radar with long-range detection capability, high resolution, and anti-interference performance. This type of radar can maintain stable detection performance in adverse weather conditions (such as fog, rain, and snow).

[0030] Configuration: Install millimeter-wave radars on the front, side, and rear of the vehicle to achieve accurate detection of obstacles in the front, side, and rear.

[0031] LiDAR: Selection: Choose a lidar with high accuracy, long-range detection capability, and high-density point cloud output. This type of radar can provide detailed three-dimensional information about the vehicle's surroundings.

[0032] Configuration: Install the LiDAR on the top of the vehicle to ensure that it can scan the surrounding environment without obstruction. At the same time, consider the scanning frequency and resolution of the LiDAR to meet the needs of different driving scenarios.

[0033] Other sensors: Other types of sensors can also be integrated as needed, such as ultrasonic sensors (for close-range obstacle detection), inertial navigation systems (for vehicle attitude and position estimation), etc.

[0034] Preprocess the collected raw data, such as denoising, filtering, calibration, etc., to improve the quality and accuracy of the data. Perform color correction and brightness adjustment on the image data to improve the visual effect and recognition rate of the image. Identify and process abnormal or erroneous data, such as discarding, re-collecting or interpolating, to avoid negative impact on subsequent processing and analysis.

[0035] The multimodal data acquisition module 110 can collect multi-dimensional information of the vehicle's surrounding environment more comprehensively and accurately, providing a solid data foundation for subsequent perception and decision-making systems. Specifically, the image data collected by the vehicle-mounted camera is input into the image data processing module 120, and the radar data is input into the radar data processing module 130 to obtain the final driving strategy.

[0036] An image data processing module 120 is used to process the image data using an improved deep learning model to extract feature information of a target object in the image; wherein the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein the network model architecture is composed of a backbone network, a Transformer encoder, a category counting module, a counting-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder; The image data processing module 120 processes the image data of the vehicle camera using the improved deep learning model to extract the features of the target objects in the image, such as vehicles, pedestrians, traffic signs, etc. Specifically, it includes the following contents: (1) Building a network model architecture The model as a whole consists of a backbone network, a Transformer encoder, a category counting module, a count-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder. The input image is fed into the backbone network to extract multi-scale features and then fed into the Transformer encoder to obtain visual features. The category counting module determines the number of object queries for the decoder, the count-guided feature enhancement module strengthens the visual features, the dynamic query selection module optimizes object queries, and the efficient hybrid decoder generates categories and bounding boxes. Flexible scaling of the model is achieved by adjusting the width and depth of the efficient hybrid encoder and the parameters of the decoder. Reducing the number of decoder layers has little effect on accuracy, but can greatly improve the inference speed, making the model suitable for different scenarios.

[0037] Preferably, the step of processing the image data using an improved deep learning model to extract feature information of the target object in the image includes: The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels; Sending the multi-scale feature map to the Transformer encoder for self-attention operation to capture long-distance dependencies in the features and generate visual features enhanced by self-attention; The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances; The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map; The dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries; The efficient hybrid decoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent target recognition and bounding box regression processing to extract feature information of target objects in the image.

[0038] The backbone network receives the image data collected by the vehicle camera as input, extracts the multi-scale features of the image, and generates a series of feature maps at different levels (such as ), these feature maps contain basic information and high-level semantic information in the image.

[0039] Transformer encoder: Data operation: Receives high-level feature maps (such as) output by the backbone network as input, performs self-attention operations on the feature maps, captures long-range dependencies in the features, and generates self-attention-enhanced visual features that focus more on important areas in the image.

[0040] (2) Building a category counting module Category counting module: receives the high-resolution feature map output by the Transformer encoder (such as ,Assuming that it is some kind of intermediate feature map output by the encoder, it may actually be a further processed feature) as input. The density map is generated by dilated convolution and the number of image instances is classified. And the number level of target objects in the image is output to provide a basis for the number of queries for the subsequent dynamic query selection module.

[0041] Specifically, the category counting module consists of a density extractor and a classification head. The density extractor performs dilated convolution on the high-resolution feature map to generate a density map, and the classification head classifies the number of image instances into four levels based on the map, avoiding the difficulty of regressing the exact number and providing quantity information for subsequent modules.

[0042] Density extractor generates density map using dilated convolution , the formula is: ;

[0043] : density map; : High-resolution feature map output by the Transformer encoder; DilatedConv: Dilated convolution is used to expand the receptive field and capture long-distance dependency information. Dilated convolution expands the receptive field by inserting holes while keeping the parameters and computational complexity unchanged. The formula is as follows: ;

[0044] in, : Output feature map Location; : Input feature map at index The value of :The convolution kernel is in The weight of each position; : Expansion rate, used to control the spacing of the voids; : The size of the convolution kernel (such as ).

[0045] The classification head classifies the number of image instances into four levels:

[0046] The classification is achieved by the following formula: ;

[0047] Among them, Linear: Linear layer is the basic building block of the fully connected layer, which is used for feature conversion or dimension change. The linear layer implements the linear mapping between input and output, and the formula is: ;

[0048] in, : input vector, is the input vector dimension; : output vector, m is the output vector dimension; : weight matrix; : bias vector.

[0049] Softmax: Used to generate classification probability P. Softmax maps the input vector to a probability distribution. The formula is: ;

[0050] in, : Input vector elements; : After normalization The probability of an element; : length of input vector.

[0051] (3) Constructing a counting-guided feature enhancement module Count-guided feature enhancement module: It receives the density map output by the category counting module and the visual features of the Transformer encoder as input, and uses the density map to enhance the visual features of the encoder through spatial cross-attention and channel attention operations to generate count-guided enhanced feature maps, which are enhanced both in space and channels, and are particularly beneficial for the detection of small objects.

[0052] Specifically, the count-guided feature enhancement module uses the density map of the category counting module , the encoder visual features are enhanced through spatial cross attention and channel attention operations, and the encoder visual features are spatially and channel-enhanced. First, the density map is sampled to generate a multi-scale count feature map, and the spatial attention map is generated through pooling and convolution operations, which is multiplied with the visual features; then the channel attention operation is performed on the spatial enhancement feature to obtain the count-guided enhancement feature map, which improves the spatial information of small objects.

[0053] More specifically, the density map is used to generate the spatial attention map , the formula is as follows: ;

[0054] in, : density map; :Respectively and convolution; , MaxP: global average pooling and maximum pooling; Concat: concatenate pooling features along the channel axis; : Sigmoid activation function, used to generate spatial attention map ; : Cross product operation; : Features after spatial enhancement; : Original multi-scale feature map; : Spatial attention map.

[0055] The above process uses the density map to generate a spatial attention map through a series of operations .

[0056] Channel attention is further applied to the spatially enhanced features, and the formula is: ;

[0057] in, : channel attention weight; : Count-guided enhanced features; MLP: Multi-layer perceptron, composed of multiple linear layers and nonlinear activation functions, performs complex nonlinear mapping on input features for feature extraction or classification. The formula is: ;

[0059] in, : The output of the multi-layer perceptron unit; : input vector; : weight matrix of the linear layer; : bias vector; : Non-linear activation function (such as ReLU or Sigmoid).

[0060] According to the above process, the count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map, including: The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations; the spatial attention map is multiplied with the original visual feature to obtain a spatially enhanced feature; a channel attention operation is performed on the spatially enhanced feature to obtain a count-guided enhanced feature map.

[0061] (4) Constructing dynamic query selection Dynamic query selection module: receives the count-guided enhanced feature map and the classification result of the category counting module as input, and is used to determine the number of queries for the decoder based on the classification result ,And optimize the query. The query consists of content and location information, generating query content and location that adapts to the image target density, providing accurate query information for the subsequent decoding process.

[0062] Enhance the feature map using count guidance and select the top features, through linear transformation and The prediction generates query content and position deviation respectively, and adjusts the query position to adapt it to the image target density, which is conducive to small target positioning.

[0063] Further, the query quantity The allocation rules are: ;

[0065] Query content and location using enhanced features generate: ;

[0066] in, : Enhancing features from counts based on classification scores Before the selection Features; Linear: The linear layer is the basic building block of the fully connected layer and is used for feature transformation or dimension change. The linear layer implements a linear mapping between input and output; The query content is generated through linear transformation; : Query the position, generate the position deviation through the feedforward network and add the anchor box bias ; : Anchor frame bias; FFN: Feedforward network, which is a key component in Transformer. Its structure is similar to that of a multilayer perceptron. It is used in Transformer to process features position by position and enhance the nonlinear modeling ability of the model. The formula is: ;

[0067] : Weight matrix.

[0068] : bias vector.

[0069] : Non-linear activation function (such as GELU or ReLU).

[0070] : The number of neurons in the hidden layer, i.e., the dimension of the hidden layer; hidden layers are the layers between the input layer and the output layer in a neural network, and they are responsible for learning the representation and features of the data. In the Transformer model, the hidden layers in a feedforward network (FFN) usually have dimensions larger than the input embedding and output embedding (i.e. ) larger dimensions (i.e. > ) to increase the nonlinear modeling capability of the model.

[0071] In the Transformer model, it refers to the dimensions of the input embedding and output embedding, and is also the dimension of all key, query, and value vectors in the model. It is a key hyperparameter in the Transformer model, affecting the expressiveness and computational complexity of the model.

[0072] (5) Building an efficient hybrid encoder Efficient hybrid encoder: receives multi-scale feature maps output by the backbone network (such as ) and the queries generated by the dynamic query selection module are used as input. Through the attention-based intra-scale feature interaction and the cross-scale feature fusion based on convolutional neural network, the richness and computational efficiency of feature representation are improved, and the fused multi-scale features are generated. These features are more accurate and efficient, providing strong support for subsequent detection tasks.

[0073] The efficient hybrid encoder is designed to include attention-based intra-scale feature interaction and cross-scale feature fusion based on convolutional neural network by analyzing the computational redundancy of multi-scale Transformer encoder. The attention-based intra-scale feature interaction performs intra-scale interaction on high-level features, which mainly serves to only Perform self-attention operations to capture the relationship between conceptual entities in high-level features while reducing the risk of duplication or invalid interaction with low-level features. Cross-scale feature fusion fuses adjacent scale features through fusion blocks, integrates adjacent scale features into richer feature representations, improves speed and accuracy, and outputs fused multi-scale features for subsequent object recognition and bounding box regression.

[0074] The calculation formula of efficient hybrid encoder is: ;

[0075] in, : Keys, queries, and values ​​are obtained by mapping the last level features Flattened generation; High-level features generated by the backbone network; : Features after Softmax processing; : The dimension of the key, used to scale the attention weight. In the self-attention mechanism, the key vector is used to calculate the similarity between the query vector and the value vector. is an important hyperparameter in the Transformer model, and Related, usually ,in, is the number of attention heads.

[0076] Softmax: Normalizes the attention scores to indicate the strength of the relationship between different positions; Flatten: Flatten the input features, that is, transform the input features into 2D tensors to adapt to the input requirements of Transformer; Reshape: restore the flattened features to their original shape, that is, restore the flat features output by Transformer to the shape of a multi-scale feature map to facilitate fusion with other features; CCFF: Cross-scale feature fusion module, combining features of multiple scales into one output , as defined below: ;

[0077] in, : Input features of adjacent scales; Conv1x1: one Convolution operation is used to adjust the number of feature channels or enhance features; Add: element-by-element addition operation, used to integrate multi-path features; RepBlock: combines convolution operation with residual connection to improve the ability and speed of feature extraction, used for feature fusion and enhancement, see the following definition for details:

[0079] in, : Input feature map; : Output feature map; Conv3x3: one Standard convolution operation to capture local spatial features; Conv1x1: a Convolution operation, used to adjust the number of feature channels or enhance features; +: element-by-element addition, combining feature representations from different paths; : Activation function (such as ReLU or SiLU) used to introduce non-linearity.

[0080] (6) Constructing loss function The total loss function of the detection task is a weighted combination of multiple losses, including L1 loss , bounding box regression loss , category loss and classification loss Weighted loss function To optimize the detection task: ; : L1 loss, used for bounding box positioning regression; : Bounding box regression loss, used for bounding box optimization; : Category loss, used to deal with category imbalance problems; : classification loss, used for classification tasks; : Weight coefficient, which controls the impact of each loss on the total loss; L1 loss : Calculate the absolute error between the predicted bounding box and the true bounding box: ; : The number of target instances; : No. A ground-truth bounding box; : No. predicted bounding boxes; : L1 norm, that is, the sum of absolute values.

[0081] Bounding Box Regression Loss : Based on the intersection-over-union ratio, the convergence speed and accuracy of bounding box regression are optimized, and the traditional intersection-over-union ratio loss is improved by introducing the area of ​​the bounding box: ;

[0082] and : The area of ​​the predicted box and the real box; : The minimum bounding box area of ​​the predicted box and the real box; : The area of ​​non-predicted and non-real boxes in the bounding box; IoU: The ratio of the intersection area of ​​the predicted box and the true box to the union area. It is a commonly used indicator in the field of object detection. It is used to measure the degree of overlap between the predicted box and the true box. The higher the IoU value, the closer the predicted box is to the true box.

[0083] : The intersection area of ​​the predicted bounding box and the true bounding box; : The union area of ​​the predicted bounding box and the true bounding box; Class loss : Used to alleviate the imbalance problem of positive and negative sample ratios: ; : predicted positive sample probability; : True category label (0 or 1); : weights of positive and negative samples; : Focusing factor, controlling the weight of difficult samples; Classification Loss : For classification tasks, the formula is: ; : Number of categories; : No. The sample in The true distribution of classes; : No. The sample in The predicted probability of the class.

[0084] The fused multi-scale features output by the efficient hybrid decoder contain rich information about the target object in the image. After further processing (such as object recognition and bounding box regression), specific target object features can be extracted from these features.

[0085] Specifically, the fused multi-scale features output by the efficient hybrid decoder are a more abstract and advanced feature representation that contains information of different scales and levels in the image. By performing operations such as target recognition and bounding box regression on these features, the target objects in the image (such as vehicles, pedestrians, traffic signs, etc.) can be accurately located and recognized, and their bounding boxes and category labels can be given.

[0086] Therefore, the network model constructed above is used to identify the collected image data and obtain the target object information. As part of the input to the radar data processing module 130, driving decisions are further optimized.

[0087] The radar data processing module 130 is used to analyze the change trend of the radar time series data and make the optimal driving decision according to different traffic scenarios in combination with the target object feature information; Preferably, the radar data processing module 130 is used to analyze the change trend of the radar time series data, and can make optimal decisions according to different traffic scenarios, such as normal driving, overtaking, turning, etc., such as control instructions such as acceleration, deceleration, and steering, including: Analyze the change trend of the radar time series data to obtain the current state of the vehicle; The target object feature information extracted by the image data processing module and the current state of the vehicle Combined to form an enhanced state ; ;

[0088] in, : The current state of the vehicle collected by the on-board radar, including speed , acceleration , direction angle , the distance to the obstacle ahead wait; : All target object information identified by the image data processing module 120, such as target categories such as vehicles, pedestrians, traffic signs, etc., the location of the target in the image, etc.

[0089] In the enhanced state Based on this, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem; Furthermore, the decision optimization problem is expressed as: ;

[0090] in, : Current enhancement status Next action The cumulative expected rewards received; : Current time step The optimal decision, i.e. the vehicle’s control instructions, such as acceleration, deceleration or steering; : A set of optional actions, including acceleration, deceleration, steering, etc.; Optimizing the cumulative expected reward through policy gradient methods , in order to select the optimal control action.

[0091] The decision optimization problem constructed by the radar data processing module 130 is expressed as selecting the optimal control action under the current enhancement state to maximize the cumulative expected reward.

[0092] Build reward functions to encourage the model to make safe and efficient decisions; Specifically, we design a reward function to encourage the model to make safe and efficient decisions. For example, we give positive rewards for avoiding collisions and planning routes reasonably, and negative rewards for dangerous driving behaviors.

[0093] Designing the reward function To encourage safe and efficient driving:

[0094] : Weight coefficient, used to balance the importance of each reward item.

[0095] : Safety rewards to encourage behaviors such as collision avoidance, e.g. ;

[0096] in, It is the distance between the vehicle and the obstacle in front. It is an important parameter in radar data and is used to evaluate the relative position relationship between the vehicle and the obstacle. is the safety distance threshold.

[0097] : Efficiency rewards, encouraging reasonable route planning and time saving, such as:

[0098] in, is the speed of the vehicle; is the target speed.

[0099] : Comfort rewards, penalizes sharp turns or rapid acceleration / deceleration, for example: ;

[0100] in, is the magnitude of the action change between adjacent time steps.

[0101] : Rewards for yielding to pedestrians or responding correctly to traffic signs, e.g. ;

[0102] Build a decision-making support mechanism to trigger preset safety strategies in emergency situations; Specifically, combined with the rule-based decision-making support mechanism, a series of emergency trigger conditions and corresponding control strategies are set in the program. For example, when it is detected that the obstacle ahead is too close and the speed is too fast, or an obstacle suddenly appears ahead, the emergency braking safety strategy is immediately triggered. The setting of rules ensures that the safety strategy is triggered first in an emergency: If the target category detected is "pedestrian", emergency braking is triggered.

[0103] If the traffic sign detected is "STOP", the parking action is triggered.

[0104] The formula is expressed as: ;

[0105] Among them, Trigger: emergency trigger signal; : Speed ​​threshold.

[0106] Once an emergency situation is triggered, the vehicle will Mandatory settings: ;

[0107] in, : Emergency control instructions (such as emergency braking); : Conventional control actions selected by reinforcement learning.

[0108] Under the above-mentioned reward function and decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

[0109] In the process of solving decision optimization problems, reinforcement learning algorithms (such as Q-learning, DQN, PPO, etc.) are used to optimize the cumulative expected rewards. . These algorithms iteratively update the policy or value function to find the optimal policy that maximizes the cumulative expected reward. In practical applications, the reinforcement learning model can be trained and verified using simulated environments or real-world test data. Through continuous trial and error and learning, the model can gradually master the optimal driving strategy in different traffic scenarios. After optimization by the reinforcement learning algorithm, the optimal control action under the current enhanced state can be obtained. , which is the final driving decision.

[0110] The optimal decision execution module 140 is used to execute the optimal decision action according to the optimal driving decision.

[0111] Specifically, the optimal decision execution module 140 first receives the optimal control action from the radar data processing module 130 , the action is acceleration, deceleration, or steering, etc. Then the optimal control action The control instructions are converted into actual control instructions for the vehicle, such as adjusting the accelerator pedal position, brake pedal position or steering wheel angle, etc. Further, the converted control instructions are sent to the vehicle control system to perform the corresponding driving operation. In the process of executing the control instructions, the optimal decision execution module 140 can also monitor the state of the vehicle and the changes in the surrounding environment, and feed back this information to the radar data processing module 130 and the image data processing module 120 so that they can adjust the decision and strategy in time.

[0112] For example, in an autonomous vehicle, when the optimal decision execution module 140 receives a "deceleration" decision, it converts the decision into a control instruction to reduce the accelerator pedal position or increase the brake pedal position, and sends it to the vehicle control system for execution. At the same time, it also monitors the vehicle's speed and the distance to the obstacle ahead, etc., to ensure the smooth progress of the deceleration operation.

[0113] In summary, a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided in the above embodiments of the present invention realizes high-precision perception and rapid response of vehicles to environmental information in complex traffic scenarios by designing efficient data fusion algorithms and real-time optimized decision-making models, improves perception accuracy and robustness as well as decision-making speed and efficiency, enhances system adaptability and scalability, reduces the risk of traffic accidents, etc., which is of great significance for promoting the development and application of intelligent connected vehicle technology.

[0114] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A multimodal perception and decision-making system for intelligent connected vehicles based on deep learning, characterized by: include: The multimodal data acquisition module is used to collect image data and radar time series data around the vehicle through the vehicle-mounted camera and vehicle-mounted radar sensor to achieve all-round perception of the vehicle's surrounding environment; An image data processing module, used to process the image data using an improved deep learning model to extract feature information of a target object in the image; wherein the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein the network model architecture is composed of a backbone network, a Transformer encoder, a category counting module, a counting-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder; A radar data processing module, used to analyze the changing trend of the radar time series data, and make optimal driving decisions according to different traffic scenarios in combination with the characteristic information of the target object; The optimal decision execution module is used to execute the optimal decision action according to the optimal driving decision.

2. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 is characterized in that: in, The step of processing the image data using the improved deep learning model to extract feature information of the target object in the image includes: The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels; Sending the multi-scale feature map to the Transformer encoder for self-attention operation to capture long-distance dependencies in the features and generate visual features enhanced by self-attention; The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances; The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map; The dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the query; The efficient hybrid decoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent target recognition and bounding box regression, so as to extract the feature information of the target object in the image.

3. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 or 2, characterized in that: in, The category counting module consists of a density extractor and a classification head; the density extractor is used to perform dilated convolution on the high-resolution feature map to generate a density map, and the classification head is used to classify the number of image instances into four levels according to the density map.

4. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 or 2, characterized in that: in, The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross attention and channel attention operations to obtain a count-guided enhanced feature map, including: The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations; Multiplying the spatial attention map with the original visual feature to obtain a spatially enhanced feature; A channel attention operation is performed on the spatially enhanced features to obtain a count-guided enhanced feature map.

5. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 3 is characterized in that: in, The dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification result of the category counting module, and optimizes the content and location of the query, including: Determine the number K of decoder queries according to the classification result of the category counting module; Using the counts to guide the enhancement of feature maps , based on the classification score, the top K features are selected, and the query content and position deviation are generated through linear transformation and FFN prediction respectively, and then the query position is adjusted to adapt it to the image target density to facilitate small target positioning.

6. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 or 2, characterized in that: in, The efficient hybrid decoder is a hybrid encoder based on attention-based intra-scale feature interaction and convolutional neural network-based cross-scale feature fusion; The efficient hybrid decoder performs in-scale interaction on high-level features output by the backbone network based on in-scale feature interaction of attention, and is used to perform self-attention operation only on high-level features to capture the relationship between conceptual entities in high-level features, while reducing the risk of repeated or invalid interaction on low-level features; Cross-scale feature fusion fuses adjacent scale features through fusion blocks, integrates adjacent scale features into a richer feature representation, and outputs the fused multi-scale features for subsequent object recognition and bounding box regression.

7. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1, characterized in that: in, The constructed loss function is a weighted combination of multiple losses, specifically including L1 loss , bounding box regression loss , category loss and classification loss Weighted loss function To optimize the detection task; the loss function for: ; in, , , , They correspond to L1 losses respectively. , bounding box regression loss , category loss and classification loss The weight coefficient of .

8. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1, characterized in that: The radar data processing module is used to analyze the change trend of the radar time series data and make the optimal driving decision according to different traffic scenarios in combination with the target object feature information, including: Analyze the change trend of the radar time series data to obtain the current state of the vehicle; Combining the target object feature information extracted by the image data processing module with the current state of the vehicle to form an enhanced state; Based on the enhanced state, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem; Build reward functions to encourage the model to make safe and efficient decisions; Build a decision-making support mechanism to trigger preset safety strategies in emergency situations; Under the reward function and the decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

9. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 8, characterized in that: The decision optimization problem constructed by the radar data processing module is expressed as selecting the optimal control action under the current enhanced state to maximize the cumulative expected reward; The decision optimization problem is expressed as: ； in, : In the current enhanced state Execute an action The cumulative expected rewards received; : The optimal decision at the current time step t, i.e., the control command of the vehicle; : A set of optional actions, including acceleration, deceleration, and steering; Optimize the cumulative expected reward by using the policy gradient method , in order to select the optimal control action.

10. The method for dynamic allocation of distributed network resources based on multiple strategies according to claim 9, characterized in that: in, The reward function is designed to encourage safe, efficient and comfortable driving behavior, including safety rewards, efficiency rewards, comfort rewards and object rewards; The decision support mechanism sets a series of emergency triggering conditions and corresponding control strategies.

Images