Automatic driving behavior prediction method and device based on multi-modal fusion, storage medium and electronic device

Abstract

The invention discloses an automatic driving behavior prediction method and device based on multi-modal fusion, a storage medium and an electronic device. The automatic driving behavior prediction method based on multi-modal fusion comprises the following steps: collecting visual data, language data and action data; inputting the visual data, the voice data and the action data into an automatic driving prediction model, and respectively generating scene semantic features, rule constraint and navigation intention features and action features; fusion features are obtained through the attention mechanism scene semantic features, the rule constraints, the navigation intention features and the action features; and short-term and long-term prediction results are generated based on the fusion features, and the automatic driving behavior is predicted based on the prediction results and confidence, that is, visual scene features, language rule information and a historical motion sequence are effectively fused, and a complete scene cognition framework is constructed. Therefore, the technical problem of low long-term prediction precision in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for predicting autonomous driving behavior based on multimodal fusion. Background Technology

[0002] One of the core capabilities of autonomous driving systems is the accurate prediction of the future behavior of dynamic targets in the surrounding environment (such as vehicles, pedestrians, and cyclists), which is crucial for ensuring driving safety. In current technologies, autonomous driving prediction models primarily rely on single-modal or dual-modal data for prediction. Visual-driven model: Collects image / video data through a camera, extracts environmental features using a convolutional neural network (CNN) or a visual Transformer, and predicts the future trajectory of the target.

[0003] LiDAR and vision fusion model: Combining the 3D spatial information of LiDAR point clouds with the semantic information of visual images to improve prediction robustness in complex scenes.

[0004] Action sequence model: Based on the target's historical trajectory data, time series features are encoded through recurrent neural networks (LSTM) or Transformers to achieve short-term trajectory extrapolation.

[0005] Existing technologies lack effective utilization of linguistic information in traffic scenarios, such as traffic sign text ("speed limit 50"), navigation instructions ("turn left 300 meters ahead"), and traffic rule texts ("yield to pedestrians at intersections"), making it difficult for models to understand the semantic logic and rule constraints of the scenario.

[0006] Currently, related technologies cannot construct a complete scene cognition framework, making it difficult to support accurate predictions in complex traffic environments. No effective solutions have yet been proposed to address these issues. Summary of the Invention

[0007] This invention provides a method, apparatus, storage medium, and electronic device for predicting autonomous driving behavior based on multimodal fusion, in order to at least solve the problem that the prior art is unable to support accurate prediction in complex traffic environments.

[0008] According to one aspect of the present invention, an autonomous driving behavior prediction method based on multimodal fusion is provided, comprising: collecting visual data, language data, and motion data; inputting the visual data, language data, and motion data into an autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features, and motion features, respectively; obtaining fused features by using an attention mechanism to analyze the scene semantic features, rule constraints and navigation intent features, and motion features; generating short-term and long-term prediction results based on the fused features; and predicting autonomous driving behavior based on the prediction results and confidence levels.

[0009] Optionally, generating scene semantic features includes: inputting the visual data into the visual feature extraction module in the autonomous driving prediction model, first extracting low-level features through a CNN extractor, then modeling spatial relationships through a visual Transformer, and then generating the scene semantic features containing target attributes and scene structure through a scene semantic encoder.

[0010] Optionally, the generation of rule constraints and navigation intent features includes: inputting the visual data into the language feature parsing module in the autonomous driving prediction model, extracting rule constraints from the traffic rule text, and generating rule constraint features; performing intent recognition on the navigation instructions and generating navigation intent features; and outputting the rule constraints and navigation intent features that fuse rule constraints and navigation intent.

[0011] Optionally, generating action features includes: inputting the action data into the action sequence encoding module in the autonomous driving prediction model; capturing time-related features of the input trajectory sequence in the action data through a self-attention mechanism and outputting temporal features; calculating the motion trend parameters of the target and generating trend features; modeling the interaction relationship between multiple targets through a graph neural network and generating interaction features; and fusing the temporal features, trend features, and interaction features through the action sequence encoding module to generate the action features.

[0012] Optionally, the step of obtaining fused features through attention mechanism scene semantic features, rule constraints, navigation intent features, and action features includes: calculating the attention weights of scene semantic features, rule constraints, and navigation intent features; extracting visual region features related to language description; establishing the association between visual scenes and action sequences; capturing the correspondence between target actions and scene changes; filtering invalid information in action sequences based on language rule constraints; and strengthening action features that conform to the rules; and aggregating scene semantic features, rule constraints, navigation intent features, and action features using a combination of weighted summation and residual connection to generate fused features.

[0013] According to another aspect of the present invention, an autonomous driving behavior prediction device based on multimodal fusion is also provided, comprising: a data acquisition module for acquiring visual data, language data, and motion data; a generation module for inputting the visual data, language data, and motion data into an autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features, and motion features, respectively; a fusion module for obtaining fused features through an attention mechanism of scene semantic features, rule constraints and navigation intent features, and motion features; and a prediction module for generating short-term and long-term prediction results based on the fused features, and predicting autonomous driving behavior based on the prediction results and confidence levels.

[0014] Optionally, the generation module includes: a first generation submodule, used to input the visual data into the visual feature extraction module in the autonomous driving prediction model, first extracting low-level features through a CNN extractor, then modeling spatial relationships through a visual Transformer, and then generating the scene semantic features containing target attributes and scene structure through a scene semantic encoder.

[0015] Optionally, the generation module includes: a second generation submodule, used to input the visual data into the language feature parsing module in the autonomous driving prediction model, extract rule constraints from the traffic rule text, generate rule constraint features; perform intent recognition on the navigation instructions, generate navigation intent features; and output the rule constraints and navigation intent features that fuse the rule constraints and navigation intent.

[0016] Optionally, the generation module includes: a third generation submodule, used to input the action data into the action sequence encoding module in the autonomous driving prediction model; capture time-related features of the input trajectory sequence in the action data through a self-attention mechanism and output temporal features; calculate the motion trend parameters of the target and generate trend features; model the interaction relationship between multiple targets through a graph neural network and generate interaction features; and fuse the temporal features, trend features, and interaction features through the action sequence encoding module to generate the action features.

[0017] According to a first aspect of the embodiments of this application, a computer-readable storage medium is provided, characterized in that the storage medium stores a computer program, wherein the computer program is configured to execute the above-described autonomous driving behavior prediction method based on multimodal fusion at runtime.

[0018] According to a first aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the above-described method for predicting autonomous driving behavior based on multimodal fusion.

[0019] In this embodiment of the invention, visual data, language data, and motion data are collected; these data are then input into an autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features, and motion features, respectively; a fusion feature is obtained by using an attention mechanism to integrate the scene semantic features, rule constraints and navigation intent features, and motion features; short-term and long-term prediction results are generated based on the fusion feature, and autonomous driving behavior is predicted based on the prediction results and confidence levels. This effectively integrates visual scene features, language rule information, and historical motion sequences to construct a complete scene cognition framework. This solves the technical problem of low long-term prediction accuracy in existing technologies. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a hardware structure block diagram of a mobile terminal based on an optional multimodal fusion-based autonomous driving behavior prediction method according to an embodiment of the present invention. Figure 2 This is a flowchart of an optional autonomous driving behavior prediction method based on multimodal fusion according to an embodiment of the present invention; Figure 3 This is an overall architecture diagram of an optional vision-language-action-based autonomous driving prediction model according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an optional visual feature extraction module according to an embodiment of the present invention; Figure 5 This is a flowchart of an optional language feature parsing module according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an optional action sequence encoding module according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the attention mechanism of an optional cross-modal fusion module according to an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the working principle of an optional prediction output module according to an embodiment of the present invention; Figure 9 This is a flowchart of a control method for an optional autonomous driving prediction model according to an embodiment of the present invention; Figure 10 This is a structural block diagram of an optional autonomous driving behavior prediction device based on multimodal fusion according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a sequence of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] The autonomous driving behavior prediction method based on multimodal fusion provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for an autonomous driving behavior prediction method based on multimodal fusion, according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal 10 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multimodal fusion-based autonomous driving behavior prediction method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0026] This embodiment also provides an autonomous driving behavior prediction method based on multimodal fusion. Figure 2 This is a flowchart of an autonomous driving behavior prediction method based on multimodal fusion according to an embodiment of the present invention, such as... Figure 2 As shown, the process of this autonomous driving behavior prediction method based on multimodal fusion includes the following steps: Step S202: Collect visual data, language data, and motion data.

[0027] Step S204: Input the visual data, the voice data, and the action data into the autonomous driving prediction model to generate scene semantic features, rule constraints, navigation intent features, and action features, respectively.

[0028] Step S206: Obtain fused features by using scene semantic features, rule constraints, navigation intent features, and action features through the attention mechanism.

[0029] Step S208: Generate short-term and long-term prediction results based on the fused features, and predict autonomous driving behavior based on the prediction results and confidence level.

[0030] The execution entity of the aforementioned autonomous driving behavior prediction method based on multimodal fusion may include, but is not limited to, intelligent devices, servers, and terminal devices.

[0031] The embodiments provided in this application involve collecting visual data, language data, and motion data; inputting these data into an autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features, and motion features, respectively; obtaining fused features through an attention mechanism; generating short-term and long-term prediction results based on the fused features; and predicting autonomous driving behavior based on the prediction results and confidence levels. This effectively integrates visual scene features, language rule information, and historical motion sequences to construct a complete scene cognition framework. This solves the technical problem of low long-term prediction accuracy in existing technologies.

[0032] Optionally, generating scene semantic features may include: inputting the visual data into the visual feature extraction module in the autonomous driving prediction model, first extracting low-level features through a CNN extractor, then modeling spatial relationships through a visual Transformer, and finally generating the scene semantic features containing target attributes and scene structure through a scene semantic encoder.

[0033] Optionally, the generation of rule constraints and navigation intent features may include: inputting the visual data into the language feature parsing module in the autonomous driving prediction model, extracting rule constraints from the traffic rule text, and generating rule constraint features; performing intent recognition on navigation instructions to generate navigation intent features; and outputting the rule constraints and navigation intent features that fuse rule constraints and navigation intent.

[0034] Optionally, generating action features may include: inputting the action data into the action sequence encoding module in the autonomous driving prediction model; capturing time-related features of the input trajectory sequence in the action data through a self-attention mechanism and outputting temporal features; calculating the motion trend parameters of the target and generating trend features; modeling the interaction relationship between multiple targets through a graph neural network and generating interaction features; and fusing the temporal features, trend features, and interaction features through the action sequence encoding module to generate the action features.

[0035] Optionally, the step of obtaining fused features through attention mechanism scene semantic features, rule constraints, navigation intent features, and action features may include: calculating the attention weights of scene semantic features, rule constraints, and navigation intent features; extracting visual region features related to language description; establishing the association between visual scenes and action sequences; capturing the correspondence between target actions and scene changes; filtering invalid information in action sequences based on language rule constraints; and strengthening action features that conform to the rules; and aggregating scene semantic features, rule constraints, navigation intent features, and action features using a combination of weighted summation and residual connection to generate fused features.

[0036] As an optional embodiment, this application also provides an autonomous driving behavior prediction method based on vision-language-action multimodal fusion. For example... Figure 3 The diagram shows the overall architecture of an autonomous driving prediction model based on vision, language, and action.

[0037] like Figure 3 As shown, it includes a multimodal data input layer 1, a visual feature extraction module 2, a language feature parsing module 3, an action sequence encoding module 4, a cross-modal fusion module 5, a prediction output module 6, a rule constraint optimization module 7, and a model training and update module 8.

[0038] In this embodiment, the modules work together to achieve accurate prediction of dynamic targets in autonomous driving scenarios.

[0039] The multimodal data input layer 1 is responsible for receiving various types of raw data. Among them, the visual data interface 11 receives RGB image sequences (resolution 1920×1080, frame rate 30fps) captured by the camera; the language data interface 12 receives traffic rule text (such as "No overtaking"), navigation instructions (such as "Turn right into the main road 500 meters ahead") and traffic sign OCR recognition results; and the motion data interface 13 receives historical trajectory data of targets detected by radar or vision (including parameters such as position, speed, and acceleration, with a sampling frequency of 10Hz).

[0040] like Figure 4 The diagram shows the structure of the visual feature extraction module. Visual feature extraction module 2 adopts an architecture combining CNN and Transformer, specifically including: CNN Feature Extractor 21: Employs a ResNet-50 network to extract features from the input image and outputs a 512×16×16 feature map to capture low-level visual features (such as edges and textures). Visual Transformer 22: Composed of 12 Transformer encoder layers, it flattens the feature map output by the CNN into sequence features (length 256), captures spatial correlation features through a self-attention mechanism, and outputs a 768-dimensional feature vector. Scene semantic encoder 23: Performs semantic parsing on the features output by the visual Transformer, identifies target types (such as vehicles, pedestrians, traffic lights), positional relationships, and scene structures (such as intersections, tunnels, parking lots) in the scene, and generates a 1024-dimensional scene semantic feature vector.

[0041] In practical work, the processing flow of the visual feature extraction module 2 is as follows: the input image first extracts the low-level features through the CNN feature extractor 21, then sends it to the visual Transformer 22 to model spatial relationships, and finally the scene semantic encoder 23 generates semantic features containing target attributes and scene structure, with the processing latency controlled within 50ms.

[0042] like Figure 5 The diagram shows the workflow of the language feature parsing module. Language feature parsing module 3 is used to process various types of language data, including: Text preprocessing unit 31: Cleans the input text (removes noisy characters), segments it, and standardizes it (e.g., unifies the format), converting unstructured text into a structured sequence; Pre-trained language model 32: It adopts a fine-tuned BERT-base model, takes a pre-processed text sequence as input, and outputs a 768-dimensional language feature vector; Rule Feature Generator 33: For traffic rule text, extract rule constraints (such as "speed limit ≤ 50km / h" and "yield priority") and generate 512-dimensional rule constraint features; Navigation intent parser 34: Recognizes the intent of navigation commands (such as "turn left", "go straight", "decelerate") and generates 256-dimensional navigation intent features.

[0043] The output of language feature parsing module 3 is a 1024-dimensional language feature vector that integrates rule constraints and navigation intent. It supports processing multiple language data simultaneously (such as parsing traffic signs and navigation instructions at the same time) with a processing latency of ≤30ms.

[0044] like Figure 6 The diagram shows the structure of the action sequence encoding module. Action sequence encoding module 4 is used to encode the target's historical trajectory data, and includes: Trajectory data preprocessing unit 41: performs outlier filtering and interpolation completion on the input historical trajectory (including position, velocity, and acceleration data of the past 3 seconds), and unifies the sampling frequency; Temporal Transformer42: It consists of 8 Transformer encoder layers. It takes a trajectory sequence (length 30) as input, captures temporal correlation features through a self-attention mechanism, and outputs 512-dimensional temporal features. Motion trend encoder 43: Calculates the target's motion trend parameters (such as velocity change rate and steering angle change) and generates 256-dimensional trend features; Interaction Feature Extractor 44: Models the interaction relationships (such as distance and relative speed) between multiple targets using a graph neural network (GNN) to generate 256-dimensional interaction features.

[0045] The output of the action sequence encoding module 4 is a 1024-dimensional action feature vector that integrates time sequence, trend and interaction information. It supports the simultaneous encoding of trajectory data of more than 10 dynamic targets, with a processing latency of ≤40ms.

[0046] like Figure 7 The diagram shows the attention mechanism of the cross-modal fusion module. The cross-modal fusion module 5 establishes semantic associations between multimodal features through the attention mechanism, including: Visual-Language Attention Submodule 51: Calculates the attention weights of visual features and linguistic features, and extracts visual region features related to linguistic description; Visual-Motion Attention Submodule 52: Establishes the association between visual scenes and action sequences, and captures the correspondence between target actions and scene changes; Language-Motion Attention Submodule 53: Based on language rule constraints, it filters invalid information in action sequences and strengthens action features that conform to the rules; Fusion Feature Aggregation Unit 54: It uses a combination of weighted summation and residual connection to aggregate the outputs of the three sub-modules and generate a 2048-dimensional fusion feature vector.

[0047] The core of the cross-modal fusion module 5 is to achieve semantic alignment between modalities through a cross-attention mechanism. For example, it associates the linguistic feature of "speed limit 50" with the vehicle speed feature in vision, and associates the "turn left" instruction with the target turning action. The attention weight visualization of the fusion process can be used for model interpretability analysis.

[0048] like Figure 8 The diagram shows the working principle of the prediction output module. Prediction output module 6 adopts a dual prediction head structure: Short-term prediction head 61: For short-term predictions of 0-2 seconds, a 3-layer fully connected network is used to output the target's future position coordinates (one point every 0.5 seconds), speed, and behavior category (such as going straight or turning left), with a prediction step size of 0.5 seconds; Long-term prediction head 62: For long-term predictions of 2-5 seconds, a Transformer decoder structure is adopted. Based on the fusion features and short-term prediction results, it outputs sparser future trajectory points (one point every 1 second) and behavioral trend predictions. Confidence assessment unit 64: Calculates the confidence score for each prediction result for reference in subsequent decision-making systems.

[0049] The prediction output module 6 also includes a rule constraint optimization module 7, which checks whether the prediction result conforms to traffic rules (such as whether it is speeding or whether it violates right-of-way rules) through the rule verification unit 71. If it does not conform, it is fine-tuned through the prediction corrector 72 to ensure the rationality of the prediction result.

[0050] like Figure 9 The flowchart shown illustrates the control method for an autonomous driving prediction model. The control method for an autonomous driving prediction model includes the following steps: S1, Data Input: Visual data, language data, and motion data are simultaneously acquired through the multimodal data input layer 1; S2, Visual Feature Extraction: Visual feature extraction module 2 processes visual data and generates scene semantic features; S3, Language Feature Analysis: Language feature analysis module 3 processes language data and generates rule constraints and navigation intent features; S4, Action Sequence Encoding: Action sequence encoding module 4 processes historical trajectories and generates action features; S5, Cross-modal fusion: Cross-modal fusion module 5 fuses visual, linguistic, and action features through an attention mechanism; S6, Prediction Output: Prediction output module 6 generates short-term and long-term prediction results and performs rule verification; S7, Result Feedback: Output the prediction results and confidence scores to the autonomous driving decision-making system, and at the same time feed the actual results back to the model training and update module 8 for model optimization.

[0051] In practical applications, the model runs on a 100ms cycle, updating the prediction results every 100ms to ensure real-time performance. The model is trained using a multimodal dataset (including visual, linguistic, and action annotations) and trained via the Adam optimizer. The loss function is a weighted sum of the mean squared error (MSE) of the predicted trajectory and the cross-entropy of the behavior categories.

[0052] In this embodiment, the cross-modal attention mechanism is as follows: In the cross-modal fusion module 5, scaled dot product attention is used to calculate the intermodal associations. For example, the formula for calculating the visual-language attention weights is: Attention (Q_v, K_l, V_l) = softmax ((Q_v K_l^T) / √d_k) V_l Where Q_v is the visual feature query vector, K_l and V_l are the key vector and value vector of the language feature, respectively, and d_k is the feature dimension.

[0053] Rule constraint optimization: The rule constraint optimization module 7 constructs a rule knowledge base based on the language feature parsing results and uses fuzzy logic reasoning to correct the prediction results. For example, when a "pedestrian crossing" sign is detected, the prediction weight of pedestrian crossing behavior is automatically increased.

[0054] Long-term prediction optimization: The long-term prediction head 62 adopts an autoregressive mechanism, which uses the prediction result of the previous step as input to feed back to the model. It combines attention masking technology to reduce error accumulation, and introduces prior knowledge of scene context (such as road structure) to constrain the prediction range.

[0055] It should be noted that the feature extraction network can be replaced by a YOLO architecture combined with Transformer, or a lighter MobileNet can be used as the backbone network of the CNN to reduce the amount of computation while ensuring accuracy. Language model replacement: The BERT model in language feature parsing module 3 can be replaced with RoBERTa or ALBERT, which are suitable for different language data scale scenarios; Fusion strategy alternatives: The cross-modal fusion module 5 can adopt an early fusion (fusion before feature extraction) or late fusion (fusion at the prediction layer) strategy. Early fusion is suitable for scenarios with strong modal correlation, while late fusion is suitable for scenarios with limited computing resources. Prediction Head Replacement: Prediction output module 6 can add an uncertainty prediction branch, outputting the probability distribution of the prediction results instead of deterministic results, thereby improving decision security; Multi-sensor expansion: It can add LiDAR data input and add a point cloud feature extraction branch to the visual feature extraction module to further improve the prediction robustness under severe weather conditions.

[0056] These alternatives can all achieve autonomous driving prediction functions through the fusion of vision, language, and action modalities, with only minor differences in computational efficiency, accuracy, or robustness, and are suitable for different autonomous driving application scenarios.

[0057] The beneficial effects of the embodiments of this application include: 1. Improve prediction accuracy: Through multimodal fusion, the short-term prediction accuracy in complex scenarios is improved by 15%-25%, and the long-term prediction error is reduced by more than 30%. 2. Enhanced rule-based reasoning ability: The introduction of a language feature parsing module enables the model to understand traffic rules and navigation instructions, improving prediction accuracy by 30% in rule-constrained scenarios; 3. Optimize dynamic adaptability: Through the correlation analysis between action sequences and visual scenes, the prediction response speed for sudden behaviors is improved by 20%, and the false negative rate is reduced by 18%; 4. Improve feature utilization: The cross-modal fusion mechanism increases feature utilization by 40% and reduces interference from redundant information; 5. Enhanced robustness: In adverse weather (rain, fog) and lighting change scenarios, prediction stability is improved by 25%, and the ability to resist interference is significantly enhanced.

[0058] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0059] This embodiment also provides an automated pressure testing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0060] Figure 10 This is a structural block diagram of an autonomous driving behavior prediction device based on multimodal fusion according to an embodiment of the present invention, such as... Figure 10 As shown, the autonomous driving behavior prediction device based on multimodal fusion includes: The acquisition module 101 is used to acquire visual data, language data, and motion data.

[0061] The generation module 103 is used to input the visual data, the voice data and the action data into the autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features and action features, respectively.

[0062] The fusion module 105 is used to obtain fused features by using scene semantic features, rule constraints, navigation intent features, and action features through attention mechanisms.

[0063] The prediction module 107 is used to generate short-term and long-term prediction results based on the fused features, and to predict autonomous driving behavior based on the prediction results and confidence level.

[0064] The embodiments provided in this application involve a data acquisition module 101 acquiring visual data, language data, and motion data; a generation module 103 inputting the visual data, language data, and motion data into an autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features, and motion features, respectively; a fusion module 105 using an attention mechanism to obtain fused features from the scene semantic features, rule constraints and navigation intent features, and motion features; and a prediction module 107 generating short-term and long-term prediction results based on the fused features, and predicting autonomous driving behavior based on the prediction results and confidence levels. This effectively integrates visual scene features, language rule information, and historical motion sequences to construct a complete scene cognition framework, thereby solving the technical problem of low long-term prediction accuracy in existing technologies.

[0065] Optionally, the generation module 103 may include: a first generation submodule, used to input visual data into the visual feature extraction module in the autonomous driving prediction model, first extracting low-level features through a CNN extractor, then modeling spatial relationships through a visual Transformer, and then generating scene semantic features containing target attributes and scene structure through a scene semantic encoder.

[0066] Optionally, the generation module 103 may include: a second generation submodule, used to input visual data into the language feature parsing module in the autonomous driving prediction model, extract rule constraints from traffic rule text, generate rule constraint features; perform intent recognition on navigation instructions, generate navigation intent features; and output rule constraint and navigation intent features that fuse rule constraints and navigation intent.

[0067] Optionally, the generation module 103 may include: a third generation submodule, used to input motion data into the motion sequence encoding module in the autonomous driving prediction model; capture temporal correlation features of the input trajectory sequence in the motion data through a self-attention mechanism and output temporal features; calculate the motion trend parameters of the target and generate trend features; model the interaction relationship between multiple targets through a graph neural network and generate interaction features; and fuse temporal features, trend features and interaction features through the motion sequence encoding module to generate motion features.

[0068] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0069] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0070] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1 collects visual data, language data, and motion data; S2, input visual data, voice data and motion data into the autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features and motion features respectively; S3 obtains fused features by using attention mechanism to analyze scene semantic features, rule constraints, navigation intent features, and action features; S4 generates short-term and long-term prediction results based on fused features, and predicts autonomous driving behavior based on the prediction results and confidence levels.

[0071] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0072] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0073] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0074] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1 collects visual data, language data, and motion data; S2, input visual data, voice data and motion data into the autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features and motion features respectively; S3 obtains fused features by using attention mechanism to analyze scene semantic features, rule constraints, navigation intent features, and action features; S4 generates short-term and long-term prediction results based on fused features, and predicts autonomous driving behavior based on the prediction results and confidence levels.

[0075] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0076] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting autonomous driving behavior based on multimodal fusion, characterized in that, include: Collect visual data, language data, and motion data; The visual data, the voice data, and the action data are input into the autonomous driving prediction model to generate scene semantic features, rule constraints, navigation intent features, and action features, respectively. By using attention mechanisms to extract scene semantic features, rule constraints, navigation intent features, and action features, fused features are obtained. Short-term and long-term prediction results are generated based on the fused features, and autonomous driving behavior is predicted based on the prediction results and confidence levels.

2. The method according to claim 1, characterized in that, Generate scene semantic features, including: The visual data is input into the visual feature extraction module of the autonomous driving prediction model. First, the low-level features are extracted by the CNN extractor, then the spatial relationships are modeled by the visual Transformer, and finally the scene semantic features containing target attributes and scene structure are generated by the scene semantic encoder.

3. The method according to claim 1, characterized in that, The generation rule constraints and navigation intent features include: The visual data is input into the language feature parsing module of the autonomous driving prediction model. For the traffic rule text, the rule constraints are extracted and rule constraint features are generated. Perform intent recognition on navigation commands to generate navigation intent features; The output is the rule constraint and navigation intent features that are fused together.

4. The method according to claim 1, characterized in that, The generated action features include: The action data is input into the action sequence encoding module in the autonomous driving prediction model. The input trajectory sequence in the action data is captured by a self-attention mechanism to capture time-related features and output time-series features. Calculate the target's motion trend parameters and generate trend features; The interaction relationships between multiple targets are modeled using graph neural networks, generating interaction features. The action features are generated by fusing the temporal features, trend features, and interaction features through the action sequence encoding module.

5. The method according to claim 1, characterized in that, The fused features obtained through the attention mechanism, scene semantic features, rule constraints, navigation intent features, and action features include: Calculate the attention weights of scene semantic features, rule constraints, and navigation intent features, and extract visual region features related to language description; Establish a connection between visual scenes and action sequences, and capture the correspondence between target actions and scene changes; Based on language rule constraints, invalid information in action sequences is filtered out, and action features that conform to the rules are strengthened. By combining weighted summation with residual connection, scene semantic features, rule constraints, navigation intent features, and action features are aggregated to generate fused features.

6. An autonomous driving behavior prediction device based on multimodal fusion, characterized in that, include: The acquisition module is used to acquire visual data, language data, and motion data; The generation module is used to input the visual data, the voice data and the action data into the autonomous driving prediction model to generate scene semantic features, rule constraints and navigation intent features and action features, respectively. The fusion module is used to obtain fused features by using scene semantic features, rule constraints, navigation intent features, and action features through an attention mechanism; The prediction module is used to generate short-term and long-term prediction results based on the fused features, and to predict autonomous driving behavior based on the prediction results and confidence levels.

7. The apparatus according to claim 6, characterized in that, The generation module includes: The first generation submodule is used to input the visual data into the visual feature extraction module in the autonomous driving prediction model. First, the low-level features are extracted by a CNN extractor, then the spatial relationships are modeled by a visual Transformer, and finally the scene semantic features containing target attributes and scene structure are generated by a scene semantic encoder.

8. The apparatus according to claim 6, characterized in that, The generation module includes: The second generation submodule is used to input the visual data into the language feature parsing module in the autonomous driving prediction model, extract rule constraints for traffic rule text, and generate rule constraint features. Perform intent recognition on navigation commands to generate navigation intent features; The output is the rule constraint and navigation intent features that are fused together.

9. The apparatus according to claim 6, characterized in that, The generation module includes: The third generation submodule is used to input the action data into the action sequence encoding module in the autonomous driving prediction model, and to capture time-related features of the input trajectory sequence in the action data through a self-attention mechanism, and output time-series features. Calculate the target's motion trend parameters and generate trend features; The interaction relationships between multiple targets are modeled using graph neural networks, generating interaction features. The action features are generated by fusing the temporal features, trend features, and interaction features through the action sequence encoding module.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 5 when it is run.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images