Visual-semantic collaborative awareness method and system for automatic driving vehicle

Abstract

The invention relates to a vision-semantic collaborative perception method and system for an automatic driving vehicle. Comprising a visual-semantic feature coding module and a visual-semantic feature fusion module. The visual-semantic feature encoding module comprises a text semantic encoder and a visual space-time encoder; a visual space-time encoder extracts single-frame semantic and structural features based on a 2D visual encoding model, and a 3D visual encoding model is adopted for modeling to form 3D frame-level fusion visual perception features; and the visual-semantic feature fusion module comprises a time converter, a cross converter and a context enrichment converter, and is used for adaptively adjusting a fusion weight based on a dynamic weight fusion strategy to generate required visual-semantic feature fusion information. According to the invention, the multi-modal weight is automatically adjusted, the environment perception error and decision delay are reduced, the adaptability and response precision of the automatic driving vehicle to the trunk line logistics complex traffic scene are improved, and the safe and stable operation of the intelligent driving system in the complex road environment is ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of automatic driving, and particularly relates to a visual-semantic collaborative perception method and system for an automatic driving vehicle. BACKGROUND

[0002] Highway trunk logistics is the core artery of maintaining the operation of the industrial chain. Under this background, the application of automatic driving technology in the point-to-point trunk highway transportation scene has become an important exploration direction to improve efficiency and safety. Compared with passenger cars, heavy trucks have characteristics such as large load, high inertia, long braking distance, and strong nonlinear dynamic response. The automatic driving technology of heavy trucks faces higher complexity of perception error accumulation, control delay and other bottlenecks, which has caused a significant lag in the development process, and the actual application is not mature. In view of the above technical bottlenecks, the end-to-end intelligent driving technology, with the advantage of deep coupling of "perception-decision-control" full link, is expected to break through the shackles of current heavy truck automatic driving technology, and provide an intelligent solution with high robustness and adaptability for heavy trucks in trunk logistics.

[0003] Environmental perception is the premise and foundation for the safe and reliable driving of an autonomous vehicle, and determines the level of vehicle intelligence. In the heavy truck scenario of trunk logistics, environmental perception faces many challenges, such as insufficient generalization in complex scenarios, traditional modular perception models rely on limited labeled data sets, and are difficult to adapt to the high-view blind area and long-tail-shaped obstacles specific to heavy trucks; perception-decision-control separation, complex weather and shaped obstacles in long-distance transportation cause the error of the perception system to accumulate step by step, making it difficult to meet the end-to-end reliability requirements; multi-modal perception is fragmented, and existing fusion methods such as feature splicing and single-layer attention mechanism do not consider the heterogeneity of visual and semantic features, which can easily cause feature conflicts in sudden changes in light and occlusion scenarios, resulting in a decrease in semantic reasoning ability. SUMMARY

[0004] The purpose of the present application is to provide a visual-semantic collaborative perception method and system for an automatic driving vehicle, which solves the problem that the traditional heavy truck driving system cannot provide accurate and efficient visual perception information for the end-to-end large model according to the actual situation in the trunk logistics scenario, resulting in target missed detection, false detection and other situations, and cannot meet the dual requirements of perception efficiency and reliability optimization in complex dynamic scenarios for new energy heavy trucks at the present stage.

[0005] The technical solution for achieving the purpose of the present application is a visual-semantic collaborative perception system for an automatic driving vehicle, comprising a visual-semantic feature encoding module and a visual-semantic feature fusion module.

[0006] The visual-semantic feature encoding module comprises a text semantic encoder and a visual spatio-temporal encoder. The text semantic encoder is based on a pre-training model CLIP and a BERT feature extraction model, and is used to extract semantic text features of a driving scene. The visual spatio-temporal encoder is based on a 2D visual encoding model, and is used to extract single-frame semantic and structural features to obtain a single-frame 2D feature sequence. The 2D visual encoding model is based on a 3D visual encoding model, and is used to form a 3D frame-level fusion visual perception feature by using a hierarchical window attention mechanism.

[0007] The visual-semantic feature fusion module comprises a time converter, a cross converter and a context enrichment converter. The time converter takes the single-frame 2D feature sequence as input, uses a multi-head attention mechanism to model across time sequences, combines a time sequence encoding and a residual connection strategy, captures motion features and environmental changes between continuous picture sequences, and outputs updated 2D visual representation features. The cross converter injects the 3D frame-level fusion visual perception feature into a text feature space of the pre-training model CLIP, and obtains a fusion feature of enhanced visual representation by using a cross-modal attention mechanism. The context enrichment converter introduces a self-attention mechanism to promote cross-modal interaction between up-flow visual and text features, optimizes the fusion feature of enhanced visual representation, and obtains an optimized context visual-semantic enhanced joint feature. The visual-semantic feature fusion module uses a dynamic weight fusion strategy to adaptively adjust fusion weights of the time converter, the cross converter and the context enrichment converter according to real-time environmental information, and generates visual-semantic feature fusion information required by the autonomous vehicle.

[0008] A method for visual-semantic collaborative perception of an autonomous vehicle based on the above system, comprising the following steps:

[0009] Step (1): inputting heavy truck trunk line logistics driving scene information into a text semantic encoder, converting the heavy truck trunk line logistics driving scene information into a natural language description by the text semantic encoder, extracting preliminary semantic text features by using a pre-training model CLIP, and extracting driving scene semantic text features by using a BERT feature extraction model;

[0010] Step (2): extracting single-frame semantic and structural features of the heavy truck trunk line logistics driving scene information by using a 2D visual encoding model to obtain a single-frame 2D feature sequence , and forming a 3D frame-level fusion visual perception feature by using a 3D visual encoding model and a hierarchical window attention mechanism ;

[0011] Step (3): taking the single-frame 2D feature sequence as input, modeling across time sequences for each frame sequence by using a multi-head attention mechanism, performing time sequence encoding and residual connection, capturing motion features and environmental changes of a perception target between different picture sequences, and obtaining a 2D visual representation feature ;

[0012] Step (4): The cross converter fuses 3D frame-level visual perception features By injecting the text feature space of the pre-trained model CLIP, and through a cross-modal attention mechanism, fused features that enhance visual representation are obtained. ;

[0013] Step (5): The context-enriched converter introduces a self-attention mechanism to promote cross-modal interaction between upstream visual and textual features, optimizes the fusion features of the enhanced visual representation, and obtains the optimized context-visual-semantic enhanced joint features. ;

[0014] Step (6): The visual-semantic feature fusion module adaptively adjusts the fusion weights of the time converter, cross converter and context-rich converter based on the real-time environment information according to the dynamic weight fusion strategy, and generates the visual-semantic feature fusion information required by the autonomous vehicle.

[0015] Furthermore, step (1) specifically involves:

[0016] Road information from heavy-duty truck long-haul logistics driving scenarios is used as the original label. This original label is then converted into a natural language description using the pre-trained model CLIP. Let the original label be... ,in Indicates the first Each label has a pre-training process as follows:

[0017]

[0018] In the formula, This represents the preprocessing function. Representing the normalized set of labels, establish the learned mapping relationship between labels and text in the data:

[0019]

[0020] In the formula, This is a template for generating tags to text. For template parameters, To maximize the probability of the corresponding Values, For the pre-processed label, The conditional probability of generating text based on tags;

[0021] By adjusting template parameters This maximizes the probability that the model will generate realistic text given the labels; specifically:

[0022]

[0023] In the formula, is the adjusted template parameter, is the first sample of output text, is the value corresponding to the minimum probability, is the model parameter Next, the probability of generating ; based on the label-text learning mapping relationship, a BERT feature extraction model is designed, and a data-driven method is constructed to generate the text description required by the downstream task:

[0024]

[0025] In the formula, is the generated text description set, is the label-to-text mapping function.

[0026] Further, step (2) is specifically:

[0027] Step (21): extract single-frame images, perform block patch segmentation, linear embedding, and global self-attention modeling on the single-frame images, and obtain the local and global feature representation of each frame, i.e. single-frame 2D feature sequence .

[0028] Step (22): The 3D visual coding model stacks the 2D feature sequence of multiple frames into a video body, and obtains the 3D frame-level fusion visual perception feature through 3D coding .

[0029] Further, step (21) is specifically:

[0030] Step (211) data preprocessing: adjust the size and normalize each frame of image to obtain the target image size , represented as:

[0031]

[0032]

[0033] In the formula represents the pixel value of the corresponding position of the original image, represents the coordinate of the corresponding position of the original image, represents the floating-point coordinate of the corresponding position of the original image.

[0034] Step (212): Perform block patch segmentation and linear embedding on single-frame images: divide the single-frame image into fixed-size non-overlapping blocks patch, set the input image size to , and the selected patch block size is​ Each patch block is regarded as a visual word; for each patch block, the initial embedding is obtained by convolution and linear projection, and the formula is:

[0035]

[0036] In the formula is the embedding vector of the th patch block, is the linear transformation matrix, is the flattening of the th patch block into a vector, is the th patch block, is the number of channels of the input image, is the embedding dimension;

[0037] The position encoding is added to each embedding vector:

[0038]

[0039] In the formula is the embedding sequence, is the position encoding vector;

[0040] Step (213) global self-attention modeling: the embedding sequence is projected by a linear layer to obtain query, key and value vectors respectively:

[0041]

[0042] In the formula is the feature sequence input to the Transformer layer, are the projection matrices of query, key and value respectively;

[0043] In each attention head , the standard self-attention score is calculated, and the multi-head results are connected and projected:

[0044]

[0045]

[0046] In the formula is the attention head score, is the projection result of the th attention head, is the normalized inner product, is the number of multi-head attentions, is the multi-head attention output projection matrix, is the multi-head attention connection and projection result;

[0047] Layer-by-layer calculation of multi-head self-attention and feedforward network processing within the layer, superposition of residual and layer normalization, to obtain the final 2D feature sequence

[0048]

[0049]

[0050]

[0051]

[0052] wherein is the normalized processing result, is a feedforward transformation for enhancing feature expression, is the processing result of the feedforward network, is the final 2D feature representation, is an intermediate activation function in the feedforward network, is the full connection layer parameter of the feedforward network.

[0053] Further, step (22) is specifically:

[0054] A video sequence containing T frames is designed, and after step (21) processing, a single-frame feature sequence is obtained , for each frame According to the rearrangement of the original spatial order of the patch block, it is stacked into video body , which is represented as follows:

[0055]

[0056]

[0057] wherein, is the number of image / video frames, is the total number of patch blocks, satisfying , represents the reconstruction row and column number of the single-frame patch block in space;

[0058] Local windows are divided in time and space dimensions, and the local window size is set to , that is, covering frames in the time dimension, and covering patch block areas in space, each window performs multi-head self-attention, and the output features of all windows are spliced and projected into new 3D space-time feature representation:

[0059] ​

[0060] wherein, is a new 3D spatio-temporal feature converted by the 3D encoding model, is a merging function, is each window feature;

[0061] fusing the 2D feature sequence with the 3D video volume into , for each frame, performing global pooling on the fused feature to obtain a 3D frame-level fused visual perception feature output for downstream tasks, is expressed as follows:

[0062]

[0063]

[0064]

[0065] wherein, is feature concatenation, , is a fused linear projection matrix.

[0066] Further, step (3) is specifically:

[0067] Step (31): inputting the single-frame 2D feature sequence into a time converter, the single-frame feature sequence is inputted into is:

[0068]

[0069] wherein, is a visual frame-level feature, is the number of frames, is the number of patch blocks in the ViT;

[0070] Step (32): using a multi-head attention mechanism to model each frame sequence across time sequences, and performing time sequence encoding and residual connection to capture the motion features and environmental changes of the perception target between different picture sequences, so as to obtain a 2D visual representation feature :

[0071] wherein the multi-head self-attention time sequence modeling is:

[0072]

[0073]

[0074]

[0075] wherein, is a multi-head self-attention mechanism, is a multi-head attention output projection matrix, is an attention head score, is a self-attention operation, is a projection matrix for query, key, value, is a query, key and value vector, is a normalized inner product;

[0076] The temporal encoding and residual connection process is:

[0077]

[0078]

[0079]

[0080] wherein, is a temporal encoding result, is a feedforward network operation process, is a feedforward network intermediate activation function, is a feedforward network fully connected layer parameter, is a 2D visual representation generated by the temporal transformer.

[0081] Further, step (4) is specifically:

[0082] Taking the 3D frame-level fused visual perception feature and the semantic text feature as the input of the cross transformer, performing cross-modal attention calculation, taking the visual as the query, and the text as the key value:

[0083]

[0084]

[0085] wherein, is a cross-modal attention mechanism output, is a query, key and value vector in the 3D frame-level fusion process;

[0086] The cross-modal attention mechanism output is subjected to interlayer fusion, residual addition and normalization processing to obtain a visual enhanced feature deeply fused with the semantic text:

[0087]

[0088]

[0089] wherein, is an intermediate feature after fusion but not subjected to the feedforward network, is layer normalization, is feedforward network processing, is the fusion feature of enhanced visual representation as the input of the next layer.

[0090] Further, step (5) is specifically:

[0091] The fusion feature of enhanced visual representation and the semantic text feature are cross-modal spliced along the sequence dimension to form a joint representation, providing a unified input for subsequent interaction:

[0092]

[0093] In the formula, is a visual-semantic joint representation sequence;

[0094] The fusion feature of enhanced visual representation and the semantic text feature are both stabilized by residual connection and layer normalization during training to obtain an enhanced joint feature:

[0095]

[0096]

[0097] In the formula, is a visual-semantic enhanced joint feature processed by an attention mechanism, is layer normalization, is a multi-head self-attention mechanism, is feedforward network processing, is a contextual visual-semantic enhanced joint feature;

[0098] The enhanced joint feature is generated into independent visual features and text features by a split function:

[0099]

[0100] In the formula, is the split-optimized visual feature and text feature, is a split function.

[0101] Further, step (6) is specifically:

[0102] Step (61): Let the input original image or preprocessed feature be I, and design an environment evaluation network MobileNetV3 to obtain an environmental feature vector :

[0103]

[0104] In the formula, For environmental assessment network, For various environmental impact factors;

[0105] Step (62): according to the environmental feature vector , the feature weight is calculated by using two-layer multilayer perception MLP processing:

[0106]

[0107]

[0108] In the formula, is the output hidden layer feature of the first layer of MLP, is the activation function, , respectively, the first layer weight matrix and the bias term, is the original weight without normalization, , respectively, the second layer weight matrix and the bias term;

[0109] The original weight is normalized:

[0110]

[0111] In the formula, respectively, the visual branch weight, the text branch weight and the context branch weight, to ensure that the weight and are 1;

[0112] Step (63): global cross-modal feature fusion is carried out:

[0113]

[0114] In the formula, is the accurate visual-semantic feature fusion information, is the global average pooling, is the optimized visual feature and the text feature, is the context visual-semantic enhanced joint feature.

[0115] Compared with the prior art, the present application has the following advantages:

[0116] The visual-semantic collaborative perception method and system for an autonomous vehicle described in the present application aims to improve the perception ability and decision reliability of the vehicle in the trunk logistics driving environment. The method fuses the complementary advantages of vision and semantic information in the space-semantic level by constructing a feature encoding mechanism of vision and semantic information, fully excavates the potential associated information in the driving scene, and significantly improves the recognition accuracy of dynamic targets and the robustness of overall scene understanding. Further, the system introduces a cross-temporal and cross-modal deep feature fusion strategy, and combines a dynamic weight self-adaptive adjustment mechanism, which can dynamically adjust the perception results according to the time-varying characteristics of the environment, optimize the perception performance, and improve the response ability and stability of the system in the trunk logistics driving scene. The collaborative perception system not only provides high-precision and high-robustness perception data support for the downstream end-to-end autonomous driving decision and control module, but also provides a solid technical foundation for realizing safe and efficient intelligent driving. BRIEF DESCRIPTION OF DRAWINGS

[0117] Figure 1 Flowchart of the visual-semantic collaborative perception method for an autonomous vehicle described in the present application.

[0118] Figure 2 Complex driving environment information of the vehicle described in the present application.

[0119] Figure 3 Schematic diagram of the internal structure of the designed visual encoder. DETAILED DESCRIPTION

[0120] The present application will be further described in detail below with reference to the accompanying drawings.

[0121] Referring to the accompanying Figure 1 The visual-semantic collaborative perception method for an autonomous vehicle described in the present application includes two steps of visual-semantic feature encoding and visual-semantic feature fusion.

[0122] S1: In the visual-semantic feature encoding process, according to the complex driving scene information of the new energy heavy truck, a text semantic encoder and a visual spatio-temporal encoder are designed respectively.

[0123] S11: The text encoder is mainly responsible for converting the input label information into natural language description, and extracting text features using a pre-trained model. The specific process is as follows:

[0124] The input original label (category number or keyword) is first converted into natural language description through preprocessing. Let the original label be wherein represents the th label (category number, keyword, etc.), and the preprocessing process is represented as:

[0125]

[0126] wherein, denotes the pre-processing function (such as cleaning, deduplication, standardization), denotes the normalized label set. Thus, the learning mapping relationship of label-text in data is established:

[0127]

[0128] wherein, is the generation template of label to text, is the template parameter, is the value corresponding to the maximum probability, is the pre-processed label, is the conditional probability of generating text based on the label. By adjusting the template parameter , the model maximizes the probability of generating real text under the given label condition. Specifically:

[0129]

[0130] wherein, is the adjusted template parameter, is the value corresponding to the minimum probability. Based on the above mapping relationship of label-text, the pre-trained language model BERT based on Transformer is designed, and the data-driven method is constructed to generate the text description required by the downstream task:

[0131]

[0132] wherein, is the generated text description set, is the mapping function of label to text. The text input label after optimization can be represented as: ["corrosive", "leakage", "highway"], and the output can be represented as: "highway area detects corrosive substance leakage, please evacuate immediately".

[0133] ​​S12: The visual spatio-temporal encoder designs a 2D visual code based on ViT, extracts the global semantics and local structure features of single-frame images based on the 2D visual code ViT. At the same time, a 3D visual coding technology Video SwinTransformer is designed and introduced, which is seamlessly integrated with the 2D visual code, and the spatio-temporal features are fused through the hierarchical window attention mechanism to capture the motion pattern of the dynamic target, and finally the text features and visual perception frame-level features are generated. Based on the single-frame feature extraction of ViT, the input single-frame image is segmented, linearly embedded (Patch Embedding), and global self-attention modeling is performed, so as to obtain the local and global feature representation of each frame. Combined with the above feature representation, the 3D visual code stacks the 2D features of multiple frames into a video body, and finally obtains the visual representation of the frame-level fused spatio-temporal semantic information by designing a 3D encoder, and the specific process is as follows:

[0134] Suppose the input image or video size is (where H is the height, W is the width, and C is the number of channels), in order to ensure that the subsequent patch block segmentation, embedding and network training can be carried out at the same scale, normalization and size adjustment preprocessing are performed for each input image. Usually, the pixel value is normalized to a fixed interval (the pixel value is mapped from [0, 255] to [0, 1] or [-1, 1] range), and the input image is unified to a fixed resolution. The normalization operation can be represented as:

[0135]

[0136] In the formula, and are the mean and standard deviation of the first channel, respectively, and represent the original image target position coordinates. The size adjustment can be represented as: let the size of the original image be , and the target size be . In order to maintain the original structure of the image, uniform scaling is adopted, the scaling factor is determined, and the new image coordinates are calculated:

[0137]

[0138]

[0139] In the formula, represent the height and width scaling factors, and represent the horizontal and vertical pixel coordinates in the target image, and represent the corresponding pixel coordinates in the original image. Based on the above formula, the target image size , which can be expressed as:

[0140]

[0141]

[0142] wherein represents the pixel value of the corresponding position of the original image, and the entire process is repeated for all target image pixels to achieve size scaling and obtain a preprocessed image.

[0143] The preprocessed image is divided into fixed-size non-overlapping patches, and each patch is regarded as a visual word. Assuming that the size of the input image is , the size of the selected patch is , and the number of patch blocks obtained is:

[0144]

[0145] wherein is the height and width of the input image. For each patch, it is flattened into a vector with a length of , and an embedding vector is obtained through linear projection mapping, and the formula is:

[0146]

[0147] wherein is the embedding vector of the th patch, is a linear transformation matrix, is the flattening of the th patch into a vector, is the number of channels of the input image, is the embedding dimension. This step is equivalent to a convolution operation with a kernel size of and a step size of , thereby realizing non-overlapping local region feature extraction. Subsequently, a position encoding is added to each embedding vector:

[0148]

[0149] wherein is a position encoding vector, which ensures that the model perceives the spatial order between patch blocks. The embedding sequence is subjected to global relationship modeling by a multi-layer Transformer encoder, which internally contains the following structure:

[0150] Multi-head attention mechanism: the input sequence is projected through a linear layer to obtain query, key and value vectors:

[0151]

[0152] where is the feature sequence input to the Transformer layer, are the projection matrices for query, key, value respectively. Each attention head Within, compute the standard self-attention score and concatenate and project the multi-head results:

[0153]

[0154]

[0155] where is the attention head score, is the projection result of the th attention head, is the normalized inner product to prevent the value from being too large and causing the normalized gradient to be too small, is the number of multi-head attention, is the multi-head attention output projection matrix, is the multi-head attention connection and projection result. Combine the above results, the multi-head attention output and input are first processed by a multi-head self-attention sublayer with residual connection and normalization (LayerNorm), and then through a feedforward network sublayer containing residual connection and normalization to complete the feature transformation. ViT stacks the above Transformer encoder L layers, after processing by each layer, the final output is a 2D feature representation:

[0156]

[0157]

[0158]

[0159]

[0160] where is the normalization processing result, is the feedforward network processing result, is the final 2D feature representation, is the intermediate activation function in the feedforward network, is the feedforward network fully connected layer parameter.

[0161] Based on the Video Swin Transformer, 3D visual coding stacks multi-frame 2D features into a video body. By designing a 3D encoder, it utilizes hierarchical window attention to model local and global information interaction in time and space, and finally obtains visual representation of frame-level fused spatio-temporal semantic information. The specific process is as follows:

[0162] For video data, the time dimension information needs to be modeled. A video sequence contains T frames. After 2D single-frame coding, a single-frame feature sequence is obtained. These features are rearranged according to the spatial position, and can be stacked into a three-dimensional body:

[0163]

[0164]

[0165] In the formula, is the number of image / video frames, is the total number of patch blocks, which satisfies , represents the reconstruction row and column number of single-frame patch blocks in space, is the embedding dimension. For the above three-dimensional body in time and space dimension modeling, the video body is divided into multiple local windows, each window covering a local area in space and a certain continuous picture sequence. The local window size is set as , that is, it covers frames in the time dimension and patch block areas in space. For the features in each window, self-attention operation is performed:

[0166]

[0167] In the formula, are the window sizes in time, vertical space and horizontal direction respectively, are the query, key and value matrices of the window features, which are obtained by local linear transformation in the window, is the relative position bias matrix, which encodes the relative position information between patch blocks in the window. The features of each window after self-attention operation are spliced, and through linear projection, they go through the Transformer encoder stack as in the 2D visual coding, and are converted into new 3D spatio-temporal features:

[0168]

[0169] In the formula, is the new 3D spatio-temporal feature converted by the Video Swin Transformer, is the merging function, For each window feature. In the last stage, the 2D semantic features are fused with the 3D spatio-temporal features, and mapped to a unified dimension using a fully connected layer. For each frame, the fused features are globally pooled to obtain new 3D frame-level features, which are uniformly represented as:

[0170]

[0171]

[0172]

[0173] In the formula, Feature splicing, , is a fused linear projection matrix, used for mapping and fusing dimensions, is the final 3D frame-level feature of the t-th frame after global pooling.

[0174] S2: In the process of visual-semantic feature fusion, a time converter, a cross converter and a context-rich converter are designed respectively, and a dynamic weight fusion strategy is proposed to obtain the visual-semantic fusion information required by the autonomous vehicle, and the specific process is as follows:

[0175] S21: The time converter takes a single-frame feature sequence as input, efficiently models the continuous acquisition of the spatio-temporal dynamic relationship between the picture sequence at the 2D feature level, captures the motion features and environmental changes, uses the multi-head attention mechanism to model each frame sequence across time, and performs time series encoding and residual connection to capture the motion features and environmental changes of the perception target between different picture sequences, to obtain the visual representation, which is represented as follows:

[0176] The single-frame feature sequence input is:

[0177]

[0178] In the formula, is the visual frame-level feature, is the number of frames, is the number of patch blocks in ViT, is the embedding dimension.

[0179] The multi-head self-attention time series modeling process is:

[0180]

[0181]

[0182]

[0183] wherein, is a multi-head self-attention mechanism, is a multi-head attention output projection matrix, is an attention head score, is a self-attention operation, is a projection matrix for queries, keys, values, are query, key and value vectors, is a normalized inner product.

[0184] Temporal encoding and residual connection process:

[0185]

[0186]

[0187]

[0188] wherein, is a temporal encoding result, is a feed-forward network operation process, is a feed-forward network intermediate activation function, is a feed-forward network fully connected layer parameter, is a 2D visual representation generated by the time converter.

[0189] S22: The cross converter injects the 3D space-time feature depth extracted by the Video Swin Transformer into the CLIP text feature space, adopts a cross-modal attention mechanism, and enhances the detection target visual representation, as follows:

[0190] Taking 3D visual space-time features and text semantic features as inputs:

[0191]

[0192]

[0193] wherein, is a 3D visual space-time frame-level feature obtained in visual space-time encoding, is a set of text descriptions generated in the text encoder. The above inputs are calculated by cross-modal attention, taking visual as query and text as key and value:

[0194]

[0195]

[0196] wherein, is a cross-modal attention mechanism output, Query, key and value vectors for 3D frame-level fusion process. Inter-layer fusion, residual addition and normalization processing are performed on the cross-modal attention calculation results to obtain visual enhanced features deeply fused with text semantics:

[0197]

[0198]

[0199] In the formula, is the intermediate feature after fusion without passing through the feedforward network, is layer normalization, is feedforward network processing, is the fusion feature of enhanced visual representation, which is input to the next layer.

[0200] S23: The context-rich converter promotes cross-modal interaction of visual and text features from the upper stream by introducing a self-attention mechanism, optimizes visual and text features, and enhances the context awareness ability of the final representation, as follows:

[0201] For the fusion feature of enhanced visual representation and the original text semantic information, cross-modal splicing is performed along the sequence dimension to form a joint representation, which provides a unified input for subsequent interaction:

[0202]

[0203] In the formula, is the visual-semantic joint representation sequence, is the fusion feature of enhanced visual representation, is the text description set generated in the text encoder. Subsequently, self-attention enhancement is performed on the visual-semantic joint representation sequence, both using residual connection and layer normalization to stabilize training, and finally the enhanced joint features are generated into independent visual and text optimization branches through the split function:

[0204]

[0205]

[0206] In the formula, is the visual-semantic enhanced joint feature processed by the attention mechanism, is layer normalization, is multi-head self-attention mechanism, is feedforward network processing, is the context visual-semantic enhanced joint feature. For the final output result, the optimized visual and text features are obtained by splitting:

[0207]

[0208] wherein, is the split optimization visual feature and text feature, is the split function

[0209] S24: The dynamic weight fusion strategy adaptively adjusts the fusion weight of each modality according to real-time environmental information, and finally generates accurate visual-semantic feature fusion information, as follows:

[0210] Let the input original image or preprocessed feature be I, and design an environment evaluation network based on MobileNetV3 to obtain an environment feature vector :

[0211]

[0212] wherein, is the environment evaluation network, is each type of environmental influence factor (light intensity score, scene target density, weather change, etc.). According to the environment feature vector , the modality feature weight is predicted and calculated to realize adaptive fusion:

[0213] After two layers of MLP (Multi-Layer Perceptron) processing, including linear transformation and ReLU activation, and normalization:

[0214]

[0215]

[0216] wherein, is the first layer output hidden layer feature of the MLP, is the activation function, , are the first layer weight matrix and bias term, respectively, is the original weight without normalization, , are the second layer weight matrix and bias term, respectively.

[0217] Normalization:

[0218]

[0219] wherein, are the visual branch weight, the text branch weight, and the context branch weight, respectively, to ensure that the weight and are 1. Finally, global cross-modal feature fusion is performed:

[0220]

[0221] wherein, for precise visual-semantic feature fusion information, for global average pooling, for optimized visual features and text features, for context visual-semantic enhanced joint features.

[0222] The above is an exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The protection scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A vision-semantic collaborative perception system for autonomous vehicles, characterized in that, It includes a visual-semantic feature encoding module and a visual-semantic feature fusion module; The visual-semantic feature encoding module includes a text semantic encoder and a visual spatiotemporal encoder. The text semantic encoder is based on the pre-trained CLIP and BERT feature extraction models and is used to extract semantic text features of driving scenarios. The visual spatiotemporal encoder is based on a 2D visual encoding model to extract semantic and structural features of a single frame and obtain a single-frame 2D feature sequence. Based on the single-frame 2D feature sequence, a 3D visual encoding model is used to form 3D frame-level fusion visual perception features using hierarchical window attention modeling. The visual-semantic feature fusion module includes a temporal transformer, a cross transformer, and a context-rich transformer. The temporal transformer takes a single-frame 2D feature sequence as input, uses a multi-head attention mechanism for cross-temporal modeling, and combines temporal coding and residual connection strategies to capture motion features and environmental changes between consecutively acquired image sequences, and outputs updated 2D visual representation features. The cross-converter injects 3D frame-level fused visual perception features into the text feature space of the pre-trained model CLIP, and obtains fused features that enhance visual representation through a cross-modal attention mechanism; The context-rich converter introduces a self-attention mechanism to facilitate cross-modal interaction between upstream visual and textual features, optimizes the fusion features of enhanced visual representations, and obtains optimized context-visual-semantic enhanced joint features. The visual-semantic feature fusion module uses a dynamic weight fusion strategy to adaptively adjust the fusion weights of the time converter, cross converter, and context-rich converter based on real-time environmental information, thereby generating the visual-semantic feature fusion information required by autonomous vehicles.

2. A method for visual-semantic collaborative perception of autonomous vehicles based on the system described in claim 1, characterized in that: Includes the following steps: Step (1): Input the heavy truck trunk logistics driving scenario information into the text semantic encoder, which converts it into a natural language description, uses the pre-trained model CLIP to extract preliminary semantic text features, and uses the BERT feature extraction model to extract the driving scenario semantic text features. Step (2): Use a 2D visual coding model to extract the semantic and structural features of single frames of heavy-duty truck trunk logistics driving scene information, and obtain the single-frame 2D feature sequence. A 3D visual coding model is used to form 3D frame-level fusion visual perception features by utilizing hierarchical window attention modeling. ; Step (3): The time converter uses a single-frame 2D feature sequence As input, a multi-head attention mechanism is used to model each frame sequence across time, and temporal encoding and residual connections are performed to capture the motion features of the perceived target and environmental changes between different image sequences, so as to obtain 2D visual representation features. ; Step (4): The cross converter fuses 3D frame-level visual perception features By injecting the text feature space of the pre-trained model CLIP, and through a cross-modal attention mechanism, fused features that enhance visual representation are obtained. ; Step (5): The context-enriched converter introduces a self-attention mechanism to promote cross-modal interaction between upstream visual and textual features, optimizes the fusion features of the enhanced visual representation, and obtains the optimized context-visual-semantic enhanced joint features. ; Step (6): The visual-semantic feature fusion module adaptively adjusts the fusion weights of the time converter, cross converter and context-rich converter based on the real-time environment information according to the dynamic weight fusion strategy, and generates the visual-semantic feature fusion information required by autonomous vehicles.

3. The method according to claim 2, characterized in that, Step (1) is as follows: Road information from heavy-duty truck long-haul logistics driving scenarios is used as the original label. This original label is then converted into a natural language description using the pre-trained model CLIP. Let the original label be... ,in Indicates the first Each label has a pre-training process: ， In the formula, This represents the preprocessing function. Representing the normalized set of labels, establish the learned mapping relationship between labels and text in the data: ， In the formula, This is a template for generating tags to text. For template parameters, To maximize the probability of the corresponding Values, For the pre-processed label, The conditional probability of generating text based on tags; By adjusting template parameters This maximizes the probability that the model will generate realistic text given the labels; specifically: ， In the formula, To adjust the template parameters, For the first A sample of the output text, To minimize the probability corresponding to Values, For model parameters Below, generate The probability of obtaining the label-text mapping is determined; based on the learning mapping relationship, the feature extraction model BERT is designed, and a data-driven method is constructed to generate the text descriptions required for downstream tasks. ， In the formula, For the generated text description set, This is a function for mapping tags to text.

4. The method according to claim 3, characterized in that, Step (2) specifically involves: Step (21): Extract single-frame images, perform patch segmentation, linear embedding, and global self-attention modeling on the single-frame images to obtain the local and global feature representations of each frame, i.e., the single-frame 2D feature sequence. . Step (22): The 3D visual coding model stacks the 2D feature sequences of multiple frames into a video volume, and obtains 3D frame-level fusion visual perception features through 3D coding. .

5. The method according to claim 4, characterized in that, Step (21) is as follows: Step (211) Data preprocessing: Resize and normalize each frame of the image to obtain the target image size. , represented as: ， In the formula This represents the pixel value at the corresponding position in the original image. This indicates the coordinates of the corresponding position in the original image. This represents the floating-point coordinates of the position in the original image corresponding to the position in the target image. Step (212) performs patch segmentation and linear embedding on a single frame image: the single frame image is segmented into non-overlapping patches of fixed size, and the input image size is set to... The selected patch block size is Each patch is treated as a visual word; for each patch, convolution and linear projection are used to obtain the initial embedding, as shown in the formula: ， In the formula For the first Embedding vectors of each patch block It is a linear transformation matrix. To make the first Each patch block is flattened into a vector. For the first A patch block The number of channels in the input image. For the embedded dimension; Add positional encoding to each embedding vector: ， In the formula For embedded sequences, This is a position-encoded vector; Step (213) Global self-attention modeling: for the embedded sequence The query, key, and value vectors are obtained by projecting them through linear layers respectively: ， In the formula The feature sequence input to the Transformer layer, These are the projection matrices for the query, key, and value, respectively. Each attention head Internally, calculate the standard self-attention score, and concatenate and project the multi-head results: ， ， In the formula To score points for attention, For the first The projection results of each attention head For normalized inner product, For multi-head attention, Output projection matrix for multi-head attention. For multi-head attention connections and projection results; Multi-head self-attention and feedforward network processing are performed layer by layer within the layer. Residuals are then stacked and layer normalization is applied to obtain the final 2D feature sequence. : ， ， ， ， In the formula For the normalized result, To enhance the feedforward transformation of feature representation, The result is the result of processing by the feedforward network. For the final 2D feature representation, For the intermediate activation function of the feedforward network, These are the parameters for the fully connected layer of the feedforward network.

6. The method according to claim 5, characterized in that, Step (22) is as follows: Design a video sequence containing T frames, and obtain a single-frame feature sequence after processing in step (21). For each frame, Rearranged according to the original space order of the patch blocks, and stacked as follows: video body , means as follows: ， ， In the formula, For image / video frame count, The total number of patch blocks, satisfying , This indicates the number of reconstructed rows and columns of a single frame patch block in space; Divide the space into local windows based on time and space dimensions, and set the size of each local window to [value]. That is, covering the time dimension Frames, spatially covering In the patch block region, multi-head self-attention is performed in each window, and the output features of all windows are concatenated and mapped to a new 3D spatiotemporal feature representation through linear projection: ， In the formula, For the new 3D spatiotemporal features transformed by the 3D encoding model, For merging functions, For each window feature; The 2D feature sequence is fused with the 3D video volume and represented in a unified manner. For each frame, global pooling is performed on the fused features to obtain the output 3D frame-level fused visual perception features for downstream tasks. , means as follows: ， ， ， In the formula, For feature splicing, , is the linear projection matrix for fusion.

7. The method according to claim 6, characterized in that, Step (3) is as follows: Step (31): Extract the 2D feature sequence of a single frame. Input time converter, single-frame feature sequence input for: ， In the formula, Visual frame-level features, For frame number, This represents the number of patch blocks in ViT; Step (32): Employ multi-head attention mechanism to model each frame sequence across time, and perform temporal encoding and residual connection to capture the motion features and environmental changes of the perceived target across different image sequences, thereby obtaining 2D visual representation features. : The multi-head self-attention temporal modeling is as follows: ， ， ， In the formula, This is a multi-head self-attention mechanism. Output projection matrix for multi-head attention. To score points for attention, For self-attention operations, The projection matrix of query, key, and value; The timing coding and residual connection process is as follows: ， ， ， In the formula, This is the result of timing coding. This refers to the feedforward network computation process. For the intermediate activation function of the feedforward network, These are the parameters for the fully connected layer of the feedforward network. A 2D visual representation generated by a time-transformer.

8. The method according to claim 7, characterized in that, Step (4) is as follows: 3D frame-level fusion of visual perception features Semantic text features are used as input to the cross-transformer for cross-modal attention computation, with visual information as the query and text as the key-value pair. ， ， In the formula, For the output of the cross-modal attention mechanism, For query, key, and value vectors in the 3D frame-level fusion process; For cross-modal attention mechanism output Inter-layer fusion, residual addition, and normalization are performed to obtain visually enhanced features that are deeply integrated with semantic text: ， ， In the formula, These are intermediate features that have not passed through the feedforward network after fusion. For layer normalization, For feedforward network processing, To enhance the fusion features of the visual representation, it serves as the input for the next layer.

9. The method according to claim 8, characterized in that, Step (5) is as follows: Fusion features for enhanced visual representation Together with semantic text features, cross-modal concatenation is performed along the sequence dimension to form a joint representation, providing a unified input for subsequent interactions: ， In the formula, It is a visual-semantic joint representation sequence; Fusion features that enhance visual representation Both semantic and textual features are trained using residual connections and layer normalization to achieve enhanced joint features: ， ， In the formula, For visual-semantic enhancement joint features processed by attention mechanism, For layer normalization, This is a multi-head self-attention mechanism. For feedforward network processing, For contextual visual-semantic enhanced joint features; The enhanced joint features are generated into independent visual and textual features by using a splitting function: ， In the formula, To separate and optimize visual and textual features, This is a splitting function.

10. The method according to claim 9, characterized in that, Step (6) specifically involves: Step (61): Let the input original image or preprocessed features be I, design the environment evaluation network MobileNetV3, and obtain the environment feature vector. : ， In the formula, For the environmental assessment network, Various environmental impact factors; Step (62): Based on the environmental feature vector The feature weights are predicted and calculated using a two-layer multilayer perceptron (MLP) processing method. ， ， In the formula, The first layer of the MLP outputs the hidden layer features. For activation function, , These are the first-layer weight matrix and the bias term, respectively. The original weights are unnormalized. , These are the second-layer weight matrix and the bias term, respectively. Normalize the original weights: ， In the formula, These are the visual branch weights, text branch weights, and context branch weights, respectively. To ensure the weight sum is 1; Step (63): Perform global cross-modal feature fusion: ， In the formula, To accurately fuse visual and semantic features, For global average pooling, To optimize the visual and text features, This is a context-visual-semantic enhanced joint feature.

Images