Multimodal perception and decision-making system for intelligent connected vehicles based on deep learning

Abstract

The present invention relates to the field of intelligent driving technology and discloses a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. The system comprises: a multimodal data acquisition module for collecting image data and radar time series data around the vehicle via an onboard camera and radar sensor to provide omnidirectional perception of the vehicle's surroundings; an image data processing module for processing the image data using an improved deep learning model to extract feature information of target objects in the image; a radar data processing module for analyzing the changing trends of the radar time series data and, in combination with the target object feature information, making optimal driving decisions based on different traffic scenarios; and an optimal decision execution module for executing optimal decision actions based on the optimal driving decisions. The present invention can achieve high-precision perception and rapid response to environmental information in complex traffic scenarios, improving driving safety and optimizing traffic efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. Background Art

[0002] With the rapid development of the global automotive industry, intelligent, connected vehicles have become a core research direction for future transportation. Traditional vehicle perception and decision-making systems primarily rely on a single type of sensor, which has limited information range and accuracy, making it difficult to meet the demand for comprehensive perception of the vehicle's surroundings in complex traffic environments. Furthermore, these systems often suffer from slow response times and low decision-making accuracy in complex scenarios such as inclement weather and traffic congestion, negatively impacting driving safety and traffic efficiency.

[0003] In recent years, deep learning technology has made significant progress in areas such as image processing, object detection, and path planning. However, current multimodal perception systems still face the following challenges: First, the accuracy of multi-sensor data fusion needs to be improved, especially in low signal-to-noise ratio or dynamic environments; second, decision models lack real-time performance and resource efficiency, making it difficult to meet the high-performance real-time processing requirements of intelligent connected vehicles.

[0004] Therefore, there is an urgent need to develop an intelligent connected vehicle perception and decision-making system that can efficiently integrate multimodal sensor data and has high-precision perception and rapid response capabilities. This is of great significance for improving driving safety and optimizing traffic efficiency. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention aims to propose a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. By designing an efficient data fusion algorithm and a real-time optimized decision-making model, the system can achieve high-precision perception and rapid response to environmental information in complex traffic scenarios, thereby improving driving safety and optimizing traffic efficiency.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning, comprising:

[0008] A multimodal data acquisition module, which collects image data and radar time series data around the vehicle through on-board cameras and radar sensors to provide omnidirectional perception of the vehicle's surroundings.

[0009] An image data processing module, configured to process the image data using an improved deep learning model to extract feature information of target objects in the image; wherein the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein the network model architecture comprises a backbone network, a Transformer encoder, a category counting module, a count-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder;

[0010] A radar data processing module is used to analyze the changing trend of the radar time series data and make optimal driving decisions based on different traffic scenarios in combination with the characteristic information of the target object;

[0011] The optimal decision execution module is used to execute the optimal decision action according to the optimal driving decision.

[0012] Preferably, the step of processing the image data using an improved deep learning model to extract feature information of the target object in the image includes:

[0013] The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels;

[0014] Feeding the multi-scale feature map into the Transformer encoder for self-attention operation to capture long-range dependencies in the features and generate visual features enhanced by self-attention;

[0015] The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances;

[0016] The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map;

[0017] The dynamic query selection module receives the count-guided enhanced feature map and determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries;

[0018] The efficient hybrid encoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent object recognition and bounding box regression.

[0019] Preferably, the category counting module consists of a density extractor and a classification head; the density extractor is used to perform dilated convolution on the high-resolution feature map to generate a density map, and the classification head is used to classify the number of image instances into four levels according to the density map.

[0020] Preferably, the count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross attention and channel attention operations to obtain a count-guided enhanced feature map, including:

[0021] The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations;

[0022] Multiplying the spatial attention map with the original visual features to obtain spatially enhanced features;

[0023] A channel attention operation is performed on the spatially enhanced features to obtain a count-guided enhanced feature map.

[0024] Preferably, the dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification result of the category counting module, and optimizes the content and location of the query, including:

[0025] Determine the number K of decoder queries according to the classification result of the category counting module;

[0026] The count guide is used to enhance the feature map F t , based on the classification score, the top K features are selected, and the query content and position deviation are generated respectively through linear transformation and FFN prediction, and then the query position is adjusted to adapt it to the image target density, which is conducive to small target positioning.

[0027] Preferably, the efficient hybrid encoder is a hybrid encoder of intra-scale feature interaction based on attention and cross-scale feature fusion based on convolutional neural network;

[0028] The efficient hybrid encoder performs intra-scale interaction on the high-level features output by the backbone network based on the intra-scale feature interaction of attention, and is used to perform self-attention operations only on the high-level features to capture the relationship between conceptual entities in the high-level features while reducing the risk of repeated or invalid interactions on low-level features.

[0029] Cross-scale feature fusion fuses adjacent scale features through fusion blocks, integrates adjacent scale features into a richer feature representation, and outputs the fused multi-scale features for subsequent target recognition and bounding box regression.

[0030] Preferably, the constructed loss function is a weighted combination of multiple losses, specifically including L1 loss L L1 , bounding box regression loss L GIoU , category loss L focal and classification loss L CE Weighted loss function L total To optimize the detection task; the loss function L total for:

[0031] L total =α1L L1 +α2L GIoU +α3L focal +α4L CE

[0032] Among them, α1, α2, α3, α4 correspond to L1 loss L L1 , bounding box regression loss L GIoU , category loss L focal and classification loss L CE The weight coefficient of .

[0033] Preferably, the radar data processing module is used to analyze the changing trend of the radar time series data and make optimal driving decisions according to different traffic scenarios in combination with the target object feature information, including:

[0034] Analyzing the changing trend of the radar time series data to obtain the current state of the vehicle;

[0035] Combining the target object feature information extracted by the image data processing module with the current state of the vehicle to form an enhanced state;

[0036] Based on the enhanced state, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem;

[0037] Build a reward function to encourage the model to make safe and efficient decisions;

[0038] Build a decision-making support mechanism to trigger preset safety policies in emergency situations;

[0039] Under the reward function and the decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

[0040] Preferably, the decision optimization problem constructed by the radar data processing module is expressed as selecting the optimal control action under the current enhanced state to maximize the cumulative expected reward;

[0041] The decision optimization problem is expressed as:

[0042]

[0043] Among them, Q(S enhanced ,a): Current enhancement state S enhanced The cumulative expected reward obtained by executing action a under t : The optimal decision at the current time step t, i.e., the vehicle control instruction; AA: A set of optional actions, including acceleration, deceleration, and steering;

[0044] The cumulative expected reward Q(S) is optimized by the policy gradient method enhanced ,a), to select the optimal control action.

[0045] Preferably, the reward function is designed to encourage safe, efficient and comfortable driving behavior, including safety rewards, efficiency rewards, comfort rewards and object rewards;

[0046] The decision support mechanism sets a series of emergency triggering conditions and corresponding control strategies.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) By integrating data from multiple sensors, including onboard cameras, millimeter-wave radars, and lidars, the system can more comprehensively and accurately perceive complex traffic environments. The improved deep learning model, through customized category counting modules and count-guided feature enhancement modules, improves the recognition and positioning accuracy of target objects, especially in the case of small and occluded targets.

[0049] (2) The real-time optimized decision model uses a highly efficient hybrid encoder and a dynamic query selection module, significantly improving decision-making speed and resource efficiency. By optimizing the cumulative expected reward using a policy gradient method, the system can quickly make optimal driving decisions in complex traffic scenarios, ensuring driving safety.

[0050] (3) The system supports the integration and configuration of multiple sensors, and can flexibly adjust the sensor type and quantity according to actual needs. By adjusting the width and depth of the efficient hybrid encoder and the parameters of the decoder, the system can achieve flexible model scaling and adapt to different scenarios and vehicle types.

[0051] (4) Design a decision-making mechanism based on reward functions and rule support to encourage the model to make safe and efficient driving behaviors. In emergency situations, the system can trigger preset safety strategies, such as emergency braking or parking, to effectively reduce the risk of traffic accidents. By rationally planning routes and reducing unnecessary stops and waiting, the system can significantly improve traffic efficiency and alleviate traffic congestion.

[0052] In summary, the deep learning-based multimodal perception and decision-making system for intelligent connected vehicles proposed in this invention has shown significant advantages in improving perception accuracy and robustness, enhancing decision-making speed and efficiency, enhancing system adaptability and scalability, reducing traffic accident risks, and improving traffic efficiency. It is of great significance to promote the development and application of intelligent connected vehicle technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0054] Figure 1 This is a unit module diagram of a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided by an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0058] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0059] This paper proposes a multimodal perception and decision-making system for intelligent connected vehicles based on deep learning. By designing an efficient data fusion algorithm and a real-time optimized decision-making model, the system achieves high-precision perception and rapid response to environmental information in complex traffic scenarios. Specific objectives include: enhancing the system's perception of target objects and traffic environments through multimodal sensor data fusion; employing an improved deep learning model, proposing a customized category counting module, a count-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder to improve the computational efficiency and reliability of the decision-making algorithm; and designing a reward-based and rule-assisted decision-making mechanism to reduce the risk of traffic accidents and ensure that vehicles can perform operations quickly and accurately in dynamic environments.

[0060] See also Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multimodal perception and decision-making system 100 for intelligent connected vehicles based on deep learning, including:

[0061] The multimodal data acquisition module 110 is used to collect image data and radar time series data around the vehicle through the vehicle-mounted camera and the vehicle-mounted radar sensor to perform omnidirectional perception of the vehicle's surrounding environment;

[0062] Multiple sensors, including onboard cameras and radar sensors (such as millimeter-wave radar and lidar), are used to provide comprehensive awareness of the vehicle's surroundings. These sensors are selected, installed, and debugged to ensure they can acquire data stably and accurately.

[0063] Sensor selection and configuration:

[0064] Car Camera:

[0065] Selection: Choose a high-resolution, wide-angle camera to ensure a broad view of the vehicle's surroundings. Also, consider the camera's night vision and anti-shake performance to improve data collection quality in inclement weather and during nighttime driving.

[0066] Configuration: Multiple cameras are installed according to different positions of the vehicle (such as front, rear, and side) to achieve all-round monitoring of the vehicle's surrounding environment.

[0067] Millimeter-wave radar:

[0068] Selection: Choose a millimeter-wave radar with long-range detection capabilities, high resolution, and anti-interference performance. This type of radar can maintain stable detection performance in adverse weather conditions (such as fog, rain, and snow).

[0069] Configuration: Millimeter-wave radars are installed on the front, side, and rear of the vehicle to achieve accurate detection of obstacles in front, on the sides, and behind.

[0070] LiDAR:

[0071] Selection: Choose a lidar with high accuracy, long-range detection capabilities, and high-density point cloud output. This type of lidar can provide detailed three-dimensional information about the vehicle's surroundings.

[0072] Configuration: Mount the LiDAR on the top of the vehicle to ensure it can scan the surrounding environment without obstruction. Also, consider the LiDAR's scanning frequency and resolution to meet the needs of different driving scenarios.

[0073] Other sensors:

[0074] Other types of sensors can also be integrated as needed, such as ultrasonic sensors (for close-range obstacle detection), inertial navigation systems (for vehicle attitude and position estimation), etc.

[0075] Perform preprocessing on the collected raw data, such as denoising, filtering, and calibration, to improve data quality and accuracy. Perform color correction and brightness adjustment on image data to improve visual quality and recognition rate. Identify and address abnormal or erroneous data, such as discarding, re-sampling, or interpolating, to avoid negative impacts on subsequent processing and analysis.

[0076] The multimodal data acquisition module 110 can collect multi-dimensional information of the vehicle's surrounding environment more comprehensively and accurately, providing a solid data foundation for subsequent perception and decision-making systems. Specifically, the image data collected by the on-board camera is input into the image data processing module 120, and the radar data is input into the radar data processing module 130 to obtain the final driving strategy.

[0077] An image data processing module 120 is configured to process the image data using an improved deep learning model to extract feature information of target objects in the image; wherein the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein the network model architecture comprises a backbone network, a Transformer encoder, a category counting module, a count-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder;

[0078] The image data processing module 120 processes the vehicle camera image data using an improved deep learning model to extract the features of target objects in the image, such as vehicles, pedestrians, traffic signs, etc. Specifically, it includes the following:

[0079] (1) Building a network model architecture

[0080] The model consists of a backbone network, a Transformer encoder, a category counting module, a count-guided feature enhancement module, a dynamic query selection module, and an efficient hybrid encoder. The backbone network extracts multi-scale features from the input image, which are then fed into the Transformer encoder to generate visual features. The category counting module determines the number of object queries in the decoder, the count-guided feature enhancement module enhances visual features, the dynamic query selection module optimizes object queries, and the efficient hybrid encoder generates categories and bounding boxes. Flexible model scalability is achieved by adjusting the width and depth of the efficient hybrid encoder and the decoder parameters. Reducing the number of decoder layers has little impact on accuracy but significantly improves inference speed, making the model suitable for different scenarios.

[0081] Preferably, the step of processing the image data using an improved deep learning model to extract feature information of the target object in the image includes:

[0082] The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels;

[0083] Feeding the multi-scale feature map into the Transformer encoder for self-attention operation to capture long-range dependencies in the features and generate visual features enhanced by self-attention;

[0084] The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances;

[0085] The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross-attention and channel attention operations to obtain a count-guided enhanced feature map;

[0086] The dynamic query selection module receives the count-guided enhanced feature map and determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries;

[0087] The efficient hybrid encoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent target recognition and bounding box regression processing to extract target object feature information in the image.

[0088] Among them, the backbone network: receives image data collected by the on-board camera as input, is used to extract multi-scale features of the image, and generates a series of feature maps at different levels (such as S3, S4, S5). These feature maps contain basic information and high-level semantic information in the image.

[0089] Transformer encoder: Data operation: Receives high-level feature maps (such as S5) output by the backbone network as input, performs self-attention operations on the feature maps, captures long-range dependencies in the features, and generates self-attention-enhanced visual features that focus more on important areas in the image.

[0090] (2) Constructing a category counting module

[0091] The category counting module receives the high-resolution feature map output by the Transformer encoder (e.g., S1, assumed to be an intermediate feature map output by the encoder, but likely a further processed feature map) as input. It generates a density map through dilated convolution and classifies the number of instances in the image. It also outputs the number of target objects in the image, providing a basis for the number of queries in the subsequent dynamic query selection module.

[0092] Specifically, the category counting module consists of a density extractor and a classification head. The density extractor performs dilated convolution on the high-resolution feature map to generate a density map. The classification head uses this map to classify the number of image instances into four levels, avoiding the difficulty of regressing the exact number and providing quantitative information for subsequent modules.

[0093] The density extractor generates a density map F using dilated convolution. c , the formula is:

[0094] F c =DilatedConv(S1),

[0095] F c : density map;

[0096] S1: high-resolution feature map output by the Transformer encoder;

[0097] DilatedConv: Dilated convolution is used to expand the receptive field and capture long-distance dependency information. Dilated convolution expands the receptive field by inserting holes while keeping the parameters and computational complexity unchanged. The formula is as follows:

[0098]

[0099] Among them, y[i]: the i-th position of the output feature map; x[i+r·k]: the value of the input feature map at index i+r·k; w[k]: the weight of the convolution kernel at the k-th position; r: the expansion rate, used to control the interval of the hole; KK: the size of the convolution kernel (for example, for 3×3 convolution, KK=9).

[0100] The classification head classifies the number of image instances N into four classes:

[0101] N∈{N≤3,3 <N≤10,10<N≤20,N> 20},

[0102] Classification is achieved through the following formula:

[0103] P=Softmax(Linear(Linear(F c )))

[0104] Among them, Linear: Linear layer is the basic building block of the fully connected layer, used for feature conversion or dimension change. The linear layer implements the linear mapping between input and output, and the formula is:

[0105] y=W·x+b

[0106] in, Input vector, d is the input vector dimension; Output vector, m is the output vector dimension; Weight matrix; Bias vector.

[0107] Softmax: Used to generate classification probability P. Softmax maps the input vector to a probability distribution. Its formula is:

[0108]

[0109] Among them, x i : the i-th element of the input vector; P i : The probability of the i-th element after normalization; n: The length of the input vector.

[0110] (3) Constructing a counting-guided feature enhancement module

[0111] Count-guided feature enhancement module: It receives the density map output by the category counting module and the visual features of the Transformer encoder as input, uses the density map to enhance the visual features of the encoder through spatial cross-attention and channel attention operations, and generates count-guided enhanced feature maps. These feature maps are enhanced both in space and channels, which is particularly beneficial for the detection of small objects.

[0112] Specifically, the count-guided feature enhancement module uses the density map F of the category counting modulec The encoder's visual features are enhanced through spatial cross-attention and channel-wise attention operations. The encoder's visual features are spatially and channel-wise enhanced. The density map is first sampled to generate a multi-scale count feature map. Pooling and convolution operations are then performed to generate a spatial attention map, which is then multiplied with the visual features. Channel-wise attention operations are then performed on the spatially enhanced features to generate a count-guided enhanced feature map, enhancing the spatial information of small objects.

[0113] More specifically, the density map is used to generate the spatial attention map W s , the formula is as follows:

[0114] W s =σ(Conv 7×7 (Concat[AvgP(Conv 1×1 (F c )),MaxP(Conv 1×1 (F c ))])),

[0115] E=W s ⊙S,

[0116] Among them, F c : density map; Conv 1×1 ,Conv 7×7 : 1×1 and 7×7 convolution respectively; AvgP, MaxP: global average pooling and maximum pooling; Concat: concatenate pooling features along the channel axis; σ: Sigmoid activation function, used to generate the spatial attention map W s ;⊙: cross product operation; E: spatially enhanced features; S: original multi-scale feature map; W s : Spatial Attention Map.

[0117] The above process uses the density map to generate the spatial attention map W through a series of operations s .

[0118] Channel attention is further applied to the spatially enhanced feature E, and the formula is:

[0119] W c =σ(MLP(AvgP(E))+MLP(MaxP(E)))

[0120] F t =W c ⊙E

[0121] Among them, W c : channel attention weight; F t: Count-guided enhanced features; MLP: Multi-layer perceptron, composed of multiple linear layers and nonlinear activation functions, performs complex nonlinear mapping on input features for feature extraction or classification. The formula is:

[0122] y=f(W2·f(W1·x+b1)+b2)

[0123] Where y is the output of the multilayer perceptron unit; x is the input vector; W1, W2 are the weight matrices of the linear layer; b1, b2 are the bias vectors; and f is the nonlinear activation function (such as ReLU or Sigmoid).

[0124] According to the above process, the count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross attention and channel attention operations to obtain the count-guided enhanced feature map, including:

[0125] The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations; the spatial attention map is multiplied by the original visual features to obtain spatially enhanced features; and a channel attention operation is performed on the spatially enhanced features to obtain a count-guided enhanced feature map.

[0126] (4) Constructing dynamic query selection

[0127] Dynamic query selection module: Receives the count-guided enhanced feature map and the classification results of the category counting module as input. It is used to determine the number of queries K for the decoder based on the classification results and optimize the query. The query consists of content and location information, generating query content and location that adapt to the image target density, providing accurate query information for the subsequent decoding process.

[0128] The count-guided enhanced feature map is used to select the top K features based on the classification score. The query content and position deviation are generated through linear transformation and FFN prediction respectively. The query position is adjusted to adapt to the image target density, which is conducive to small target positioning.

[0129] Furthermore, the allocation rule for the query quantity K is:

[0130]

[0131] The query content and location utilize enhanced features F t generate:

[0132] Q content =Linear(TopK(F t ))

[0133] Q position=FFN(TopK(F t ))+b anchor

[0134] Among them, TopK(F t ): Enhanced feature F from count guidance based on classification score t Select the first K features from the dataset; Linear: The linear layer is the basic building block of the fully connected layer and is used for feature transformation or dimension change. The linear layer implements a linear mapping between input and output; Q content : Query content, generated by linear transformation; Q position : Query the position, generate the position deviation through the feedforward network and add the anchor box bias b anchor ; b anchor : Anchor box bias; FFN: Feedforward network, which is a key component in Transformer. Its structure is similar to that of a multilayer perceptron. It is used in Transformer to process features position by position and enhance the nonlinear modeling ability of the model. The formula is:

[0135] FFN(x)=f(W2·f(W1·x+b1)+b2)

[0136] Weight matrix.

[0137] b1,b2: bias vectors.

[0138] f: non-linear activation function (such as GELU or ReLU).

[0139] d hidden : The number of neurons in the hidden layer, i.e., the dimension of the hidden layer; the hidden layer is the layer between the input layer and the output layer in the neural network, which is responsible for learning the representation and features of the data. In the Transformer model, the hidden layer in the feedforward network (FFN) usually has a dimension larger than the input embedding and output embedding (i.e., d model ) larger dimension (ie d hidden >d model ) to increase the nonlinear modeling capability of the model.

[0140] d model : In the Transformer model, it refers to the dimension of the input embedding and output embedding, and is also the dimension of all key, query, and value vectors in the model. It is a key hyperparameter in the Transformer model, affecting the model's expressive power and computational complexity.

[0141] (5) Building an efficient hybrid encoder

[0142] Efficient hybrid encoder: It receives the multi-scale feature maps (such as S3, S4, and S5) output by the backbone network and the queries generated by the dynamic query selection module as input. Through attention-based intra-scale feature interaction and cross-scale feature fusion based on convolutional neural networks, it improves the richness of feature representation and computational efficiency, and generates fused multi-scale features. These features are more accurate and efficient, providing strong support for subsequent detection tasks.

[0143] By analyzing the computational redundancy of multi-scale Transformer encoders, the efficient hybrid encoder is designed to incorporate attention-based intra-scale feature interaction and convolutional neural network-based cross-scale feature fusion. Attention-based intra-scale feature interaction performs intra-scale interaction on high-level features, primarily applying self-attention only to high-level features S5 to capture the relationships between conceptual entities in high-level features while reducing the risk of duplicate or invalid interactions with lower-level features. Cross-scale feature fusion fuses features from adjacent scales through a fusion block, integrating them into a richer feature representation for improved speed and accuracy. The fused multi-scale features are then output for subsequent object recognition and bounding box regression.

[0144] The calculation formula of efficient hybrid encoder is:

[0145] Q=K=V=Flatten(S5),

[0146]

[0147] O=CCFF({S3,S4,F5}),

[0148] Among them, Q, K, V: key, query and value, are generated by flattening the last level feature S5;

[0149] S3, S4, S5: high-level features generated by the backbone network;

[0150] F5: Features after Softmax processing;

[0151] d k : The dimension of the key, used to scale the attention weight. In the self-attention mechanism, the key vector is used to calculate the similarity between the query vector and the value vector. k It is an important hyperparameter in the Transformer model and is related to d model Related, usually Among them, num heads is the number of attention heads.

[0152] Softmax: Normalizes the attention scores to indicate the strength of the relationship between different positions;

[0153] Flatten: Flatten the input features, that is, transform the input features into 2D tensors to adapt to the input requirements of Transformer;

[0154] Reshape: restore the flattened features to their original shape, that is, restore the flat features output by Transformer to the shape of a multi-scale feature map to facilitate fusion with other features;

[0155] CCFF: Cross-scale feature fusion module, which combines features of multiple scales into one output O. See the following definition for details:

[0156]

[0157] in, Input features of adjacent scales; Conv1x1: a 1×1 convolution operation used to adjust the number of feature channels or enhance features; Add: an element-by-element addition operation used to integrate multi-path features; RepBlock: combines convolution operations with residual connections to improve the ability and speed of feature extraction, and is used for feature fusion and enhancement. See the following definitions for details:

[0158] RepBlock = σ(Conv3x3(F in )+Conv1x1(F in )+F in ),

[0159] Among them, F in : Input feature map; F out : Output feature map; Conv3x3: A 3×3 standard convolution operation used to capture local spatial features; Conv1 x1: A 1×1 convolution operation used to adjust the number of feature channels or enhance features; +: Element-wise addition to combine feature representations from different paths; σ: Activation function (such as ReLU or SiLU) used to introduce nonlinearity.

[0160] (6) Constructing the loss function

[0161] The total loss function of the detection task is a weighted combination of multiple losses, including L1 loss L L1 , bounding box regression loss L GIoU , category loss L focal and classification loss L CE Weighted loss function L total To optimize the detection task:

[0162] L total=α1L L1 +α2L GIoU +α3L focal +α4L CE

[0163] L L1 : L1 loss, used for bounding box positioning regression;

[0164] L GIoU : Bounding box regression loss, used for bounding box optimization;

[0165] L focal : Category loss, used to deal with category imbalance problems;

[0166] L CE : classification loss, used for classification tasks;

[0167] α1, α2, α3, α4: weight coefficients, controlling the impact of each loss on the total loss;

[0168] L1 loss L L1 : Calculate the absolute error between the predicted bounding box and the true bounding box:

[0169]

[0170] NN: the number of target instances;

[0171] The nth ground-truth bounding box;

[0172] The nth predicted bounding box;

[0173] ∥·∥1: L1 norm, that is, the sum of absolute values.

[0174] Bounding box regression loss L GIoU : Based on the intersection-over-union (IoU) loss, the convergence speed and accuracy of bounding box regression are optimized, and the traditional IoU loss is improved by introducing the area of the bounding box:

[0175]

[0176] A and B: the area of the predicted box and the true box;

[0177] C: The minimum bounding box area of the predicted box and the real box;

[0178] |C\(A∪B)|: the area of non-predicted and non-true boxes in the bounding box;

[0179] IoU: The ratio of the intersection area of the predicted box and the true box to the union area. It is a commonly used indicator in the field of target detection. It is used to measure the degree of overlap between the predicted box and the true box. The higher the IoU value, the closer the predicted box is to the true box.

[0180] A∩B: The intersection area of the predicted bounding box and the true bounding box;

[0181] A∪B: The union area of the predicted bounding box and the true bounding box;

[0182] Class loss L focal : Used to alleviate the imbalance problem of positive and negative sample ratios:

[0183]

[0184] p: predicted probability of positive samples;

[0185] y: true category label (0 or 1);

[0186] g t : weights of positive and negative samples;

[0187] γ: focusing factor, controlling the weight of difficult samples;

[0188] Classification loss L CE : For classification tasks, the formula is:

[0189]

[0190] H: number of categories;

[0191] y nn,h : The true distribution of the nth sample in the hth class;

[0192] The predicted probability of the nth sample in the hth class.

[0193] The fused multi-scale features output by the efficient hybrid encoder contain rich information about the target object in the image. After further processing (such as target recognition and bounding box regression), specific target object features can be extracted from these features.

[0194] Specifically, the fused multi-scale features output by the efficient hybrid encoder are a more abstract and advanced feature representation that incorporates information from different scales and layers of the image. By performing operations such as object recognition and bounding box regression on these features, target objects (such as vehicles, pedestrians, and traffic signs) in the image can be accurately located and identified, and their bounding boxes and category labels can be assigned.

[0195] Therefore, the target object information T nn As part of the input to the radar data processing module 130, driving decisions are further optimized.

[0196] Radar data processing module 130, configured to analyze the changing trend of the radar time series data and make optimal driving decisions based on different traffic scenarios in combination with the target object feature information;

[0197] Preferably, the radar data processing module 130 is used to analyze the changing trend of the radar time series data and make optimal decisions, such as acceleration, deceleration, steering and other control instructions, according to different traffic scenarios, such as normal driving, overtaking, turning, etc., including:

[0198] Analyzing the changing trend of the radar time series data to obtain the current state of the vehicle;

[0199] The target object feature information T extracted by the image data processing module nn and the current state S of the vehicle collected Combined to form enhanced state S enhanced ;

[0200] S enhanced ={S collected ,T1,T2,…,T NN}

[0201] Among them, S collected : The current state of the vehicle collected by the on-board radar, including speed v, acceleration v a , direction angle θ, distance D of the obstacle in front, etc.; T1, T2,…, T NN : All target object information identified by the image data processing module 120, such as target categories such as vehicles, pedestrians, traffic signs, etc., and the location of the target in the image.

[0202] In the enhanced state S enhanced Based on this, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem;

[0203] Furthermore, the decision optimization problem is expressed as:

[0204]

[0205] Among them, Q(S enhanced ,a): Current enhancement state S enhancedThe cumulative expected reward obtained by executing action a at the current time step t; at: the optimal decision at the current time step t, that is, the control instruction of the vehicle, such as acceleration, deceleration or steering; AA: an optional set of actions, including acceleration, deceleration, steering, etc.

[0206] Optimize the cumulative expected reward Q(S) by the policy gradient method enhanced ,a), to select the optimal control action.

[0207] The decision optimization problem constructed by the radar data processing module 130 is expressed as selecting the optimal control action under the current enhanced state to maximize the cumulative expected reward.

[0208] Build a reward function to encourage the model to make safe and efficient decisions;

[0209] Specifically, a reward function is designed to encourage the model to make safe and efficient decisions. For example, positive rewards are given for avoiding collisions and planning routes appropriately, while negative rewards are given for dangerous driving behaviors.

[0210] Design reward function R(S enhanced ,a) to encourage safe and efficient driving:

[0211] R(S enhanced ,a)=λ1R safe +λ2R efficient +λ3R comfort +λ4R object ,

[0212] λ1,λ2,λ3,λ4: weight coefficients used to balance the importance of each reward item.

[0213] R safe : Safety rewards to encourage behaviors like collision avoidance, e.g.

[0214]

[0215] Where D is the distance between the vehicle and the obstacle in front, which is an important parameter in radar data and is used to evaluate the relative position relationship between the vehicle and the obstacle; D safe is the safety distance threshold. R efficient : Efficiency rewards, encouraging reasonable route planning and time saving, such as:

[0216] R efficient =-|vv desired |,

[0217] Where v is the speed of the vehicle; v desired is the target speed.

[0218] R comfort: Comfort bonus, penalizes sharp turns or rapid acceleration / deceleration, for example:

[0219] R comfort =-|Δa|

[0220] Where Δa is the magnitude of the change in motion between adjacent time steps.

[0221] R object : Rewarding behaviors such as yielding to pedestrians or responding correctly to traffic signs, e.g.

[0222]

[0223] Build a decision-making support mechanism to trigger preset safety policies in emergency situations;

[0224] Specifically, in combination with a rule-based decision-making support mechanism, a series of emergency trigger conditions and corresponding control strategies are set in the program. For example, when an obstacle is detected too close and too fast, or an obstacle suddenly appears in front, the emergency braking safety strategy is immediately triggered. The rule setting ensures that the safety strategy is triggered first in emergency situations:

[0225] If the target category detected is "pedestrian", emergency braking is triggered.

[0226] If the traffic sign detected is "STOP", the parking action is triggered.

[0227] The formula is expressed as:

[0228]

[0229] Among them, Trigger: emergency trigger signal; v threshold : Speed threshold.

[0230] Once an emergency situation is triggered, the vehicle will take action t Will be forced to set:

[0231]

[0232] Among them, a emergency : Emergency control command (such as emergency braking); a RL : Conventional control actions selected by reinforcement learning.

[0233] Under the above-mentioned reward function and decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

[0234] In the process of solving decision optimization problems, reinforcement learning algorithms (such as Q-learning, DQN, PPO, etc.) are used to optimize the cumulative expected reward Q(S enhanced,a). These algorithms iteratively update the policy or value function to find the optimal policy that maximizes the cumulative expected reward. In practical applications, reinforcement learning models can be trained and validated using simulated environments or real-world test data. Through continuous trial and error and learning, the model can gradually master the optimal driving strategy in different traffic scenarios. After optimization by the reinforcement learning algorithm, the optimal control action a in the current enhanced state can be obtained. t , which is the final driving decision.

[0235] The optimal decision execution module 140 is configured to execute an optimal decision action according to the optimal driving decision.

[0236] Specifically, the optimal decision execution module 140 first receives the optimal control action a from the radar data processing module 130 t , the action is acceleration, deceleration, or steering, etc. Then the optimal control action a t These signals are converted into actual vehicle control commands, such as adjusting the accelerator pedal position, brake pedal position, or steering wheel angle. Furthermore, the converted control commands are sent to the vehicle control system to execute the corresponding driving operations. While executing these control commands, the optimal decision execution module 140 can also monitor the vehicle's status and changes in the surrounding environment, and feed this information back to the radar data processing module 130 and the image data processing module 120 so that they can adjust their decisions and strategies in a timely manner.

[0237] For example, in an autonomous vehicle, when the optimal decision execution module 140 receives a "deceleration" decision, it converts this decision into a control instruction to decrease the accelerator pedal position or increase the brake pedal position, and sends it to the vehicle control system for execution. Simultaneously, it monitors status information such as vehicle speed and the distance to obstacles ahead to ensure the smooth deceleration operation.

[0238] In summary, the multimodal perception and decision-making system for intelligent connected vehicles based on deep learning provided by the above-mentioned embodiments of the present invention realizes high-precision perception and rapid response of vehicles to environmental information in complex traffic scenarios by designing efficient data fusion algorithms and real-time optimized decision-making models, thereby improving perception accuracy and robustness as well as decision-making speed and efficiency, enhancing system adaptability and scalability, and reducing the risk of traffic accidents. This is of great significance for promoting the development and application of intelligent connected vehicle technology.

[0239] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A multimodal perception and decision-making system for intelligent connected vehicles based on deep learning, characterized by: include: A multimodal data acquisition module, which collects image data and radar time series data around the vehicle through on-board cameras and radar sensors to provide omnidirectional perception of the vehicle's surroundings. An image data processing module is used to process the image data using an improved deep learning model to extract feature information of target objects in the image; wherein, the construction of the improved deep learning model includes: constructing a network model architecture and a loss function; wherein, the network model architecture is composed of a backbone network, a Transformer encoder, a category counting module, a counting-guided feature enhancement module, a dynamic query selection module and an efficient hybrid encoder; the constructed loss function is a weighted combination of multiple losses, specifically including L1 loss L L1 , bounding box regression loss L GIoU , category loss L focal and classification loss L CE Weighted loss function L total To optimize the detection task; The step of processing the image data using the improved deep learning model to extract feature information of the target object in the image includes: The backbone network receives the original image data as input to perform multi-scale feature extraction and generate a series of multi-scale feature maps at different levels; Feeding the multi-scale feature map into the Transformer encoder for self-attention operation to capture long-range dependencies in the features and generate visual features enhanced by self-attention; The category counting module receives the high-resolution feature map output by the Transformer encoder as input, generates a density map through dilated convolution, and classifies the number of image instances; The count-guided feature enhancement module uses the density map generated by the category counting module to perform spatial and channel enhancement on the visual features of the Transformer encoder through spatial cross attention and channel attention operations to obtain a count-guided enhanced feature map, including: The density map is sampled to generate a multi-scale count feature map, and a spatial attention map is generated through pooling and convolution operations; Multiplying the spatial attention map with the original visual features to obtain spatially enhanced features; Performing a channel attention operation on the spatially enhanced features to obtain a count-guided enhanced feature map; The dynamic query selection module receives the count-guided enhanced feature map and determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries; The efficient hybrid encoder receives the high-level feature map output by the backbone network and the query generated by the dynamic query selection module as input, and outputs fused multi-scale features for subsequent target recognition and bounding box regression processing to extract feature information of target objects in the image; A radar data processing module is used to analyze the changing trend of the radar time series data and make optimal driving decisions based on different traffic scenarios in combination with the characteristic information of the target object; The optimal decision execution module is used to execute the optimal decision action according to the optimal driving decision.

2. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 is characterized in that: in, The category counting module consists of a density extractor and a classification head; the density extractor is used to perform dilated convolution on the high-resolution feature map to generate a density map, and the classification head is used to classify the number of image instances into four levels according to the density map.

3. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 2 is characterized in that: in, The dynamic query selection module receives the count-guided enhanced feature map, determines the number of decoder queries based on the classification results of the category counting module, and optimizes the content and location of the queries, including: Determine the number K of decoder queries according to the classification result of the category counting module; The count guide is used to enhance the feature map F t , based on the classification score, the top K features are selected, and the query content and position deviation are generated respectively through linear transformation and FFN prediction, and then the query position is adjusted to adapt it to the image target density, which is conducive to small target positioning.

4. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1 is characterized in that: in, The efficient hybrid encoder is a hybrid encoder based on attention-based intra-scale feature interaction and convolutional neural network-based cross-scale feature fusion; The efficient hybrid encoder performs intra-scale interaction on the high-level features output by the backbone network based on the intra-scale feature interaction of attention, and is used to perform self-attention operations only on the high-level features to capture the relationship between conceptual entities in the high-level features while reducing the risk of repeated or invalid interactions on low-level features. Cross-scale feature fusion fuses adjacent scale features through fusion blocks, integrates adjacent scale features into a richer feature representation, and outputs the fused multi-scale features for subsequent target recognition and bounding box regression.

5. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1, characterized in that: in, Constructed loss function L total for: L total =α1L L1 +α2L GIoU +α3L focal +α4L CE Among them, α1, α2, α3, α4 correspond to L1 loss L L1 , bounding box regression loss L GIoU , category loss L focal and classification loss L CE The weight coefficient of .

6. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 1, characterized in that: The radar data processing module is used to analyze the changing trend of the radar time series data and make optimal driving decisions based on different traffic scenarios in combination with the target object feature information, including: Analyzing the changing trend of the radar time series data to obtain the current state of the vehicle; Combining the target object feature information extracted by the image data processing module with the current state of the vehicle to form an enhanced state; Based on the enhanced state, the optimal decision according to different traffic scenarios is expressed as a decision optimization problem; Build a reward function to encourage the model to make safe and efficient decisions; Build a decision-making support mechanism to trigger preset safety policies in emergency situations; Under the reward function and the decision-making assistance mechanism, the decision optimization problem is solved to obtain the final driving decision.

7. The multimodal perception and decision-making system for intelligent connected vehicles based on deep learning according to claim 6, characterized in that: The decision optimization problem constructed by the radar data processing module is expressed as selecting the optimal control action under the current enhanced state to maximize the cumulative expected reward; The decision optimization problem is expressed as: Among them, Q(S enhanced ,a): Current enhancement state S enhanced The cumulative expected reward obtained by executing action a under t : The optimal decision at the current time step t, i.e., the vehicle control instruction; AA: A set of optional actions, including acceleration, deceleration, and steering; The cumulative expected reward Q(S) is optimized by the policy gradient method enhanced ,a), to select the optimal control action.

8. The method for dynamic allocation of distributed network resources based on multiple strategies according to claim 7, characterized in that: in, The reward function is designed to encourage safe, efficient and comfortable driving behavior, including safety rewards, efficiency rewards, comfort rewards and object rewards; Reward function R(S enhanced ,a) is: R(S enhanced ,a)=λ1R safe +λ2R efficient +λ3R comfort +λ4R object , λ1,λ2,λ3,λ4: weight coefficients used to balance the importance of each reward item; R safe : Safety rewards, encouraging behaviors such as collision avoidance: Where D is the distance between the vehicle and the obstacle in front, which is an important parameter in radar data and is used to evaluate the relative position relationship between the vehicle and the obstacle; D safe is the safety distance threshold; R efficient : Efficiency rewards, encouraging reasonable route planning and time saving: R efficient =―|v―v desired |, Where v is the speed of the vehicle; v desired is the target speed; R comfort : Comfort rewards, punishes sharp turns or rapid acceleration / deceleration: R comfort =―|Δa| Where Δa is the amplitude of the action change between adjacent time steps; R object : Rewards for yielding to pedestrians or responding correctly to traffic signs: The decision support mechanism sets a series of emergency triggering conditions and corresponding control strategies.

Images