Method and apparatus for using large model based on multi-modal data, and target vehicle

Abstract

This invention relates to the field of large-scale model technology, specifically to a method, apparatus, and target vehicle for using large-scale models based on multimodal data. The method involves acquiring multimodal data of the target vehicle based on a target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data; extracting target modality data from the multimodal data according to the business type corresponding to actual business needs, and constructing target prompt words; performing feature fusion on the multimodal data to obtain target fusion features; and inputting the target prompt words and target fusion features into a large language model to output the task results corresponding to the target prompt words. This eliminates the need for independent modeling for a single task, reducing development and iteration costs; simultaneously, the large model possesses high-level semantic understanding capabilities, improving the interpretability of the results and the credibility of decision-making.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and specifically to a method, apparatus, and target vehicle for using large models based on multimodal data. Background Technology

[0002] Current mainstream intelligent driving systems adopt a hierarchical modeling system of "perception-localization-prediction-planning," where each sub-module requires independent model development and training processes for specific tasks. While this approach offers strong engineering controllability, it also has significant limitations: task fragmentation leads to repetitive development and high iteration costs; fragmented input information makes it difficult to integrate deep relationships between multi-source data (images, point clouds, chassis status, high-definition maps, etc.); and it heavily relies on finely labeled data. Furthermore, its generalization ability and high-level semantic understanding (such as scene interpretation and behavior scoring) are insufficient, thus restricting the system's interpretability and decision credibility.

[0003] In recent years, Visual Language Models (VLMs) have demonstrated outstanding advantages in tasks such as cross-modal reasoning and general semantic expression, thanks to their unified graph and text representation space and Prompt instruction mechanism, providing a new path for "unified model-driven" intelligent driving.

[0004] However, VLM still faces many challenges in directly adapting to driving scenarios, including compatibility with non-textual modalities (point cloud, chassis data), transformation of driving tasks to language paradigms, semantic transfer design of the Prompt mechanism, and in-depth evaluation of scene understanding. A systematic solution has not yet been formed.

[0005] Therefore, the existing hierarchical modeling paradigm is no longer sufficient to meet the needs of complex scenarios, and there is an urgent need for a method for using large models based on multimodal data. Summary of the Invention

[0006] This invention provides a method, apparatus, and target vehicle for using large models based on multimodal data, in order to solve the problem that existing hierarchical modeling paradigms are no longer sufficient to meet the needs of complex scenarios.

[0007] In a first aspect, the present invention provides a method for using a large model based on multimodal data. The method includes: acquiring multimodal data of a target vehicle collected based on a target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data; determining the current business type according to actual business needs; generating a prompting framework corresponding to a target prompt word based on the current business type; the prompting framework includes role positioning, rules, and constraints; extracting target modal data from the multimodal data based on the prompting framework; inputting the target modal data into the prompting framework to generate a target prompt word; performing semantic recognition on the multimodal data to extract semantic features corresponding to each modality; extracting target semantic features from the semantic features corresponding to each modality based on the target prompt word; performing feature fusion on the target semantic features to generate target fused features; and inputting the target prompt word and the target fused features into a large language model to output the task result corresponding to the target prompt word.

[0008] In one optional implementation, the target modal data is input into the prompting framework to generate target prompt words, including: converting the target modal data into structured descriptive features; inputting the structured descriptive features into the prompting framework to generate initial prompt words; determining the task constraints and output requirements corresponding to the initial prompt words according to the current business type; and completing the initial prompt words based on the task constraints and output requirements to generate target prompt words.

[0009] In one optional implementation, feature fusion is performed on the target semantic features to generate target fused features, including: selecting homologous modal semantic features from the target semantic features; performing spatiotemporal attention aggregation on the homologous modal semantic features to obtain unified semantic features; selecting similar modal semantic features from the target semantic features that are similar to the unified semantic features; the similar modal semantic features and the unified semantic features are both temporal semantic features; performing cross-modal attention alignment on the unified semantic features and similar modal semantic features to obtain backup semantic features; determining other semantic features in the target semantic features as heterologous modal semantic features corresponding to the backup semantic features; fusing the backup semantic features with the heterologous modal semantic features to generate candidate semantic features; and fusing the candidate semantic features with the semantic features of the prompt words corresponding to the target prompt words to generate target fused features.

[0010] In one optional implementation, the homologous modal semantic features are driving scene image semantic features and driving scene video semantic features. Spatiotemporal attention aggregation is performed on the homologous modal semantic features to obtain unified semantic features, including: calculating the temporal attention matrix corresponding to the driving scene video semantic features; calculating the temporal aggregation features corresponding to the driving scene video semantic features based on the temporal attention matrix; obtaining spatial attention weight maps corresponding to the driving scene image semantic features and the temporal aggregation features respectively; determining the first core target region corresponding to the driving scene image semantic features based on the spatial attention weight map corresponding to the driving scene image semantic features; determining the second core target region corresponding to the temporal aggregation features based on the spatial attention weight map corresponding to the temporal aggregation features; calculating the spatial semantic mask overlap degree corresponding to the first core target region and the second core target region; aligning the first core target region and the second core target region based on the spatial semantic mask overlap degree; and fusing the aligned first core target region and the second core target region to generate unified semantic features.

[0011] In one optional implementation, the similar modal semantic features are driving scene point cloud semantic features. Cross-modal attention alignment is performed on the unified semantic features and the similar modal semantic features to obtain backup semantic features, including: mapping the unified semantic features to the corresponding 3D space under the vehicle coordinates to obtain the first position code of the unified semantic features under the vehicle coordinates; mapping the driving scene point cloud semantic features to the corresponding 3D space under the vehicle coordinates to obtain the second position code of the driving scene point cloud semantic features under the vehicle coordinates; calculating the semantic similarity between the unified semantic features and the similar modal semantic features; calculating the position similarity between the first position code and the second position code; and fusing the unified semantic features and the similar modal semantic features based on the semantic similarity and position similarity to obtain backup semantic features.

[0012] In one optional implementation, unified semantic features and similar modal semantic features are fused based on semantic similarity and positional similarity to obtain alternative semantic features, including: calculating the fusion weight information corresponding to unified semantic features and similar modal semantic features based on semantic similarity and positional similarity respectively;

[0013] Based on the fusion weight information, unified semantic features and similar modal semantic features are fused to obtain alternative semantic features.

[0014] In one optional implementation, the heterogeneous modal semantic features include structured graph features corresponding to the high-definition map and chassis operating status temporal features. The alternative semantic features are fused with the heterogeneous modal semantic features to generate candidate semantic features. This includes: inputting the structured graph features and alternative semantic features into a preset graph neural network attention module to output target-related map features; fusing the chassis operating status temporal features and alternative semantic features to obtain target-interactive chassis features; and performing cross-attention fusion on the target-related map features, target-interactive chassis features, and alternative semantic features to generate candidate semantic features.

[0015] In one optional implementation, the structured graph features and alternative semantic features are input into a preset graph neural network attention module to output target-related map features. This includes: transforming the structured graph features corresponding to the high-definition map to generate map node features corresponding to the map nodes; inputting the map node features and alternative semantic features into the preset graph neural network attention module; the preset graph neural network attention module extracting target geometric information corresponding to the target objects from the alternative semantic features; the target geometric information includes the location, size, and type of the target objects; and calculating the Euclidean distance between each map node and each target object.

[0016] Based on Euclidean distance, target map nodes whose Euclidean distance to each target object is less than a preset Euclidean distance threshold are extracted from each map node; based on Euclidean distance, the spatial weight corresponding to each target map node is determined; the category semantic correlation between each target map node and each target object is calculated; based on category semantic correlation, the semantic weight corresponding to each target map node is determined; based on spatial weight and semantic weight, the node weight corresponding to each target map node is determined; based on the node weight, the target map nodes are weighted and fused to output target-related map features.

[0017] In one optional implementation, the chassis operating state temporal features and standby semantic features are fused to obtain target interactive chassis features, including: extracting target dynamic features corresponding to the target object from the standby semantic features; the target dynamic features include the relative position change and target speed change of the target object; aligning the chassis operating state temporal features and the target dynamic features corresponding to the target object in time and space, and labeling the aligned chassis operating state temporal features and target dynamic features to generate multi-frame chassis temporal feature-target dynamic feature pairs; wherein the labeling dimensions include vehicle dynamics state, target dynamic state, and interaction type; determining the risk feature similarity corresponding to each frame chassis temporal feature-target dynamic feature pair based on the interaction type; calculating temporal self-attention weights based on the risk feature similarity to obtain the temporal self-attention weights corresponding to each frame chassis temporal feature-target dynamic feature pair; and weighting and fusing each frame chassis temporal feature-target dynamic feature pair based on the temporal self-attention weights to generate target interactive chassis features.

[0018] In one optional implementation, cross-attention fusion is performed on target-related map features, target-interactive chassis features, and backup semantic features to generate candidate semantic features, including: determining the backup semantic features as a first query vector; concatenating the target-related map features and the target-interactive chassis features to determine a first key vector; concatenating the target-related map features and the target-interactive chassis features to determine a first value vector; calculating a cross-attention weight matrix based on the first query vector and the first key vector; multiplying the cross-attention weight matrix by the first value vector to obtain preliminary semantic features; and fusing the preliminary semantic features with the backup semantic features to generate candidate semantic features.

[0019] In one optional implementation, the candidate semantic features are fused with the semantic features of the prompt word corresponding to the target prompt word to generate target fused features, including: determining the number of attention heads based on the number of core semantic dimensions corresponding to the prompt word semantic features; determining the prompt word semantic features as the second query vector; determining the candidate semantic features as the second key vector and the second value vector; and for each attention head, based on the similarity between the second query vector and the second key vector.

[0020] Secondly, the present invention provides a device for using large models based on multimodal data, the device comprising:

[0021] The acquisition module is used to acquire multimodal data of the target vehicle based on the target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data.

[0022] The module is used to determine the current business type based on actual business needs; generate a prompt framework corresponding to the target prompt words based on the current business type; the prompt framework includes role positioning, rules, and constraints; extract target modality data from multimodal data based on the prompt framework; and input the target modality data into the prompt framework to generate target prompt words.

[0023] The output module is used to perform semantic recognition on multimodal data and extract the semantic features corresponding to each modality; based on the target prompt word, it extracts the target semantic features from the semantic features corresponding to each modality; it performs feature fusion on the target semantic features to generate target fusion features; it inputs the target prompt word and the target fusion features into the large language model and outputs the task result corresponding to the target prompt word.

[0024] Thirdly, the present invention provides a target vehicle, including an electronic device and a vehicle body, wherein the electronic device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the large model usage method based on multimodal data as described in the first aspect or any corresponding embodiment above.

[0025] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method for using large models based on multimodal data according to the first aspect or any corresponding embodiment described above.

[0026] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the method for using large models based on multimodal data according to the first aspect or any corresponding embodiment described above.

[0027] The method, apparatus, and target vehicle for using a large model based on multimodal data provided in this application integrate at least two types of data, including images, videos, point clouds, chassis status, and high-definition maps. This breaks the limitations of traditional single-modal input, enriches the semantic dimension of the scene (e.g., point clouds supplement 3D geometric information, and chassis data provide dynamic state), and provides comprehensive data support for subsequent deep understanding and reasoning. Based on the business type corresponding to actual business needs, target modal data is extracted from the multimodal data to construct target prompt words. Instructions are designed specifically based on business needs (e.g., obstacle recognition, safety scoring), abstracting specific tasks into a language paradigm understandable by the large model. This achieves precise binding of "task requirements - data semantics," preventing model reasoning from deviating from the core objective and improving the targeting of task execution. Then, leveraging the cross-modal reasoning and general semantic expression capabilities of the pre-set large model, structured results (labels, scores, etc.) are directly output without the need for independent modeling for a single task, reducing development and iteration costs. Simultaneously, the large model possesses high-level semantic understanding capabilities, improving the interpretability of the results and the credibility of the decision. Attached Figure Description

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the first process of a method for using a large model based on multimodal data according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of a second process for using a large model based on multimodal data according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the third process of using a large model based on multimodal data according to an embodiment of the present invention;

[0032] Figure 4 This is a design diagram of a multi-layer, multi-source feature adapter according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the overall framework according to an embodiment of the present invention;

[0034] Figure 6 This is a design diagram of the semantic framework for intelligent driving multi-task prompts according to an embodiment of the present invention;

[0035] Figure 7This is a structural block diagram of a large model application device based on multimodal data according to an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0039] According to an embodiment of the present invention, a method for using large models based on multimodal data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a method for using large models based on multimodal data, which can be applied to electronic devices in a target vehicle. Figure 1 This is a flowchart of a method for using large models based on multimodal data according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0041] Step S101: Obtain multimodal data of the target vehicle based on the target driving scenario.

[0042] The multimodal data includes at least two of the following: driving scene image data, driving scene video data, driving scene point cloud data, target vehicle chassis operating status data, and high-definition map data.

[0043] Specifically, electronic devices can collect driving scene image data based on vehicle-mounted cameras (front-view, surround-view, and rear-view). The driving scene image data includes static visual information such as road targets (vehicles, pedestrians, and non-motorized vehicles), traffic facilities (traffic lights, road signs, and lane lines), and environmental details (road conditions and buildings).

[0044] Electronic devices can capture driving scene video data composed of continuous frames of images based on vehicle-mounted cameras (similar to image acquisition devices). The driving scene video data includes temporal information such as target movement trajectory (e.g., vehicle movement, pedestrian movement) and scene changes (e.g., traffic light switching, lane merging), capturing the dynamic characteristics of the target (speed, direction of movement) and the temporal correlation of the scene (e.g., the causal relationship of the vehicle in front decelerating → the red light turning on).

[0045] Electronic devices can collect point cloud data of driving scenes based on LiDAR. The point cloud data of driving scenes includes the 3D geometric information of the target (length, width, height, distance), spatial position coordinates, surface contours, etc., which makes up for the distance measurement error of visual data and accurately provides the 3D size and spatial distance of the target (such as the actual distance to the vehicle in front, the height of the obstacle), and is suitable for complex lighting scenes.

[0046] Electronic devices can collect target vehicle chassis operating status data based on the vehicle's CAN bus and sensors (vehicle speed sensor, acceleration sensor, etc.). The target vehicle chassis operating status data includes time-series data such as dynamic parameters (speed, acceleration, braking status, steering angle, throttle opening) and equipment status (engine speed, tire pressure), which reflect the vehicle's own operating status and driving operations (such as rapid acceleration, emergency braking, and smooth steering). It is the core basis for driving behavior assessment and safety analysis.

[0047] Electronic devices can acquire high-definition map data based on a pre-loaded high-precision map library and real-time location updates. The high-definition map data includes road topology (number of lanes, turning rules), traffic rules (speed limits, no-entry, right-of-way), road attributes (slope, curvature), and the location of fixed facilities (traffic lights, road sign coordinates).

[0048] Step S102: Based on the business type corresponding to the actual business needs, extract target modal data from the multimodal data and construct target prompt words.

[0049] Specifically, electronic devices can determine the prompt word framework based on actual business needs, and then fill in the prompt word framework based on multimodal data to construct target prompt words.

[0050] This step will be explained in detail below.

[0051] Step S103: Perform feature fusion on the multimodal data to obtain target fusion features; input the target prompt words and target fusion features into the large language model, and output the task result corresponding to the target prompt words.

[0052] Specifically, the electronic device can input the target prompt word into the large language model, and then the large language model can identify the target prompt word and output the task result corresponding to the target prompt word.

[0053] This step will be explained in detail below.

[0054] This embodiment provides a method for using a large model based on multimodal data. It integrates at least two types of data, including images, videos, point clouds, chassis status data, and high-definition maps, breaking the limitations of traditional single-modal input and enriching the semantic dimensions of the scene (e.g., point clouds supplement 3D geometric information, chassis data provides dynamic state), providing comprehensive data support for subsequent deep understanding and reasoning. Based on the business type corresponding to actual business needs, target modal data is extracted from the multimodal data to construct target prompts. Instructions are designed specifically based on business needs (e.g., obstacle recognition, safety scoring), abstracting specific tasks into a language paradigm understandable by the large model. This achieves precise binding between "task requirements" and "data semantics," preventing model reasoning from deviating from the core objective and improving the targeting of task execution. Then, leveraging the cross-modal reasoning and general semantic expression capabilities of the large language model, structured results (labels, scores, etc.) are directly output without the need for independent modeling for a single task, reducing development and iteration costs. Simultaneously, the large model possesses high-level semantic understanding capabilities, improving the interpretability of the results and the credibility of the decision.

[0055] This embodiment provides a method for using large models based on multimodal data, which can be applied to electronic devices in a target vehicle. Figure 2 This is a flowchart of a method for using large models based on multimodal data according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0056] Step S201: Obtain multimodal data of the target vehicle based on the target driving scenario.

[0057] The multimodal data includes at least two of the following: driving scene image data, driving scene video data, driving scene point cloud data, target vehicle chassis operating status data, and high-definition map data.

[0058] Please refer to the above description of step S101 for details on this step, which will not be repeated here.

[0059] Step S202: Based on the business type corresponding to the actual business needs, extract target modal data from the multimodal data and construct target prompt words.

[0060] Specifically, step S202 above may include the following steps:

[0061] Step S2021: Determine the current business type based on actual business needs.

[0062] The current business types include tag construction, driving behavior understanding and scoring, multi-source information matching degree assessment, traffic scenario and rule understanding, and data quality assessment.

[0063] Specifically, electronic devices can receive actual business requirements input by users. Then, core keywords are extracted from these requirements, such as "score driving operations," with the core keywords "score" and "driving operations," which are matched with "driving behavior understanding and scoring"; or "check whether image and point cloud data are consistent," with the core keywords "multi-source data" and "consistency," which are matched with "multi-source information matching degree assessment."

[0064] Then, the electronic device can confirm whether the selected business type covers all requirements to avoid omissions. For example, if "it is necessary to generate scene labels and evaluate the quality of label data," it is necessary to split the business type into two and generate corresponding prompt words for each. Finally, the clearly defined current business type (such as "driving behavior understanding and scoring") serves as the core basis for subsequent steps.

[0065] Step S2022: Generate a prompt framework corresponding to the target prompt word based on the current business type.

[0066] The prompt framework includes role positioning, rules, and constraints.

[0067] Specifically, electronic devices can select the corresponding professional identity based on the business type, ensuring that the identity matches the business scenario (e.g., using "expert" or "evaluator" for assessment-related businesses, and "annotator" or "quality inspector" for annotation-related businesses). Then, the core business requirements are transformed into quantifiable and actionable judgment criteria, avoiding vague expressions (e.g., instead of saying "smooth driving," say "acceleration ≤ 0.5 m / s²"). 2 Clearly define input restrictions (e.g., "use only the provided data"), output format restrictions (e.g., "retain one decimal place for the score"), and expression standardization restrictions (e.g., "clear label categorization") to ensure that the model output meets business requirements. Finally, generate a standardized prompt framework (e.g., a prompt framework for "driving behavior understanding and scoring") that is adapted to the current business type, consisting of "role positioning + rules + constraints".

[0068] Step S2023: Extract the target modality data from the multimodal data according to the prompt framework.

[0069] Specifically, electronic devices can extract the "required data dimensions," i.e., target modal data, from the rules of the prompting framework. For example, the "driving behavior scoring" rule involves acceleration and steering angle, so the acceleration / steering angle data of the chassis modality needs to be extracted. Then, valid data is filtered according to the "data extraction criteria," and abnormal data (such as blurry videos or abrupt chassis data) is removed.

[0070] Then, the electronic device converts the filtered target modal data into a "format recognizable by the cue frame," such as visual data labeled as "<front_video_feat_1-30> (Forward-looking 30-second video features), chassis data is labeled as "<chassis_feat> (3-second speed / acceleration / steering angle curve)

[0071] Step S2024: Input the target modality data into the prompt frame to generate the target prompt word.

[0072] Specifically, step S2024 above may include the following steps:

[0073] Step a1: Convert the target modal data into structured descriptive features.

[0074] Specifically, the electronic device can extract core parameters from the target modal data (such as maximum / average speed, number / location of targets). Then, it groups the data by "modal type" or "information function" (such as "vehicle dynamics information" and "environmental visual information"). Next, the electronic device can use natural language to concatenate the core parameters, highlighting key information based on business needs (e.g., highlighting "whether there was rapid acceleration" for scoring, and highlighting "target location / behavior" for tagging), and checking whether the description conforms to the principles of "clear classification, accurate information, and no redundancy," ensuring the model can quickly identify key data. Finally, it outputs "structured descriptive features" (module-based, including key parameters) organized by business type, such as the structured description of chassis data for "driving behavior scoring" mentioned above.

[0075] An example of a structured description of different modal data can be shown in Table 1 below.

[0076] Table 1. Examples of structured descriptions of data from different modalities.

[0077]

[0078] Step a2: Input the structured description features into the prompting frame to generate initial prompt words.

[0079] Specifically, the electronic device can naturally embed the "structured description features" generated in step a1 according to the structure of the prompting framework (role positioning + rules + constraints) to form a preliminary complete instruction of "role + rules + constraints + data", that is, the initial prompt words.

[0080] Step a3: Based on the current business type, determine the task constraints and output requirements corresponding to the initial prompt words.

[0081] Specifically, the electronic device can extract "must-meet boundary conditions" (task constraints) and "output format specifications" (output requirements) from the core requirements of the current business type. Referring to the prompting framework rules: task constraints must be consistent with the prompting framework rules (e.g., if the prompting framework requires "quantified scoring," task constraints must clearly specify "scoring dimensions and calculation methods"), using quantifiable and specific expressions (e.g., "retain one decimal place" and "stand alone on a separate line"), avoiding vague requirements (e.g., "clear format" and "sufficient justification"). The electronic device generates a list of "task constraints + output requirements" that precisely matches the current business type.

[0082] Step a4: Based on task constraints and output requirements, complete the initial prompt words to generate target prompt words.

[0083] Specifically, the electronic device can replace the "task instruction" section of the initial prompt (such as "Please analyze the driver's operating habits and complete the relevant evaluation task") with "task constraints + output requirements + specific task" to ensure logical coherence. Use guiding phrases such as "Please strictly follow the following task constraints and output requirements to complete the evaluation" and "Please follow the following output specifications" to clearly inform the model of the details it must adhere to. List the task constraints and output requirements in bullet points (using numbers or bullet points) to improve readability and facilitate rapid model recognition.

[0084] For example, the initial prompt ends with the task instruction: "Please analyze the driver's operating habits and complete the relevant assessment task." The supplementary task constraints are: scoring is based solely on the provided structured data, without adding subjective assumptions; both stability and safety scoring dimensions are indispensable; reasons must correspond to specific data (such as speed and acceleration), avoiding generalities. The output requirements are: operating habit analysis → scoring results → detailed reasons; scores are retained to one decimal place (0-10 points); reasons are explained across dimensions (stability reasons + safety reasons), with each reason linked to specific data.

[0085] For example, a. The target prompt corresponding to the label construction is: You are a driver. You can now see continuous front and rear view videos, large BEV (Bird's EyeView) images, maps, and other information. Based on the information you see, please provide all the scene labels you can understand, including but not limited to static elements (road type, road signs, obstacles, etc.), vehicle behavior (such as lane changing, acceleration, etc.), and interactive behavior (such as avoiding, overtaking, meeting oncoming traffic, etc.).

[0086] b. The target prompt for understanding and scoring driving behavior is: You are a driving safety assessor. You can now see the forward-looking video, vehicle speed curve, steering angle change curve, map speed limit information, and surrounding traffic flow information for the past 30 seconds. Please analyze the driver's operating habits based on this information and give a smoothness score and a safety score (out of 10), and briefly explain your reasoning.

[0087] c. The target prompt for the multi-source information matching degree assessment should be: You are an autonomous driving system diagnostician. You can now see the forward-looking video, BEV bird's-eye view, 3D point cloud detection results, and high-precision map positioning information at the same time. Please compare whether the positions, sizes, and quantities of vehicles, pedestrians, and road signs in the video and point cloud detection results are consistent, calculate the overall matching degree percentage, and indicate the objects with differences and the types of differences.

[0088] d. The target prompt for understanding traffic scenarios and traffic rules is: You are a traffic rule examiner. You can now see real-time forward-looking video, a BEV bird's-eye view, map speed limits and lane information, as well as your own vehicle's speed, acceleration, and location trajectory. Please judge whether your vehicle's driving behavior in the current scenario complies with traffic regulations, explain the specific reasons, and provide suggestions for improvement if there are any violations.

[0089] e. The target prompt for data quality assessment is: You are an autonomous driving data quality inspector. You can now see the current batch of front-view video, 3D point cloud, map matching results, and GNSS positioning data. Please conduct a comprehensive quality assessment of the data based on video clarity, point cloud density, map matching accuracy, and positioning accuracy, giving a quality score of 0-100, and listing the main influencing factors.

[0090] Step S203: Perform feature fusion on the multimodal data to obtain target fusion features; input the target prompt words and target fusion features into the large language model, and output the task result corresponding to the target prompt words.

[0091] Please refer to the above description of step S103 for details on this step, which will not be repeated here.

[0092] This embodiment provides a method for using a large model based on multimodal data. It determines the current business type and clarifies the core task direction (such as detection, scoring, and decision-making), providing precise guidance for subsequent prompt word construction and feature extraction, and preventing the process from deviating from business requirements. Based on the current business type, it generates prompt boxes (including role positioning, rules, and constraints) corresponding to the target prompt words, thereby standardizing the prompt word structure, ensuring the large model clearly defines its own positioning and execution boundaries, and improving the standardization and accuracy of reasoning. Then, it filters target modal data based on business requirements (e.g., for scoring tasks, chassis + point cloud data is prioritized) to avoid redundant data interference and improve processing efficiency and semantic focus. Non-textual modalities (images, point clouds, etc.) are transformed into structured information that the large model can understand, building a "data-language" bridge to facilitate cross-modal semantic alignment. Combining the framework and structured features, the prompt words serve both task instructions and data support, avoiding reasoning bias caused by ambiguous instructions. It determines task constraints and output requirements, clarifying the result format (such as JSON, scoring range) and execution restrictions (such as compliance verification) to ensure standardized and usable output results. Based on task constraints and output requirements, it completes the initial prompt words to generate target prompt words. Improve the completeness and relevance of prompts to achieve deep integration of "business needs - data characteristics - execution rules" and maximize the adaptability of large model tasks.

[0093] This embodiment provides a method for using large models based on multimodal data, which can be applied to electronic devices in a target vehicle. Figure 3 This is a flowchart of a method for using large models based on multimodal data according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0094] Step S301: Obtain multimodal data of the target vehicle based on the target driving scenario.

[0095] The multimodal data includes at least two of the following: driving scene image data, driving scene video data, driving scene point cloud data, target vehicle chassis operating status data, and high-definition map data.

[0096] Please refer to the above description of step S201 for details on this step, which will not be repeated here.

[0097] Step S302: Based on the business type corresponding to the actual business needs, extract target modal data from the multimodal data and construct target prompt words.

[0098] Please refer to the above description of step S202 for details on this step, which will not be repeated here.

[0099] Step S303: Perform feature fusion on the multimodal data to obtain target fusion features; input the target prompt words and target fusion features into the large language model, and output the task result corresponding to the target prompt words.

[0100] Specifically, step S303 above may include the following steps:

[0101] Step S3031: Perform semantic recognition on the multimodal data and extract the semantic features corresponding to each modality.

[0102] Specifically, electronic devices can be equipped with high-precision timing modules (such as PTP / BeiDou timing) for all sensors (cameras, LiDAR, GPS / IMU, CAN bus) to ensure that the raw data acquisition carries a unified standard timestamp (accurate to the millisecond level). The time offset of each sensor (e.g., 20ms delay for cameras, 10ms delay for LiDAR) is recorded, and a time calibration table is established.

[0103] Electronic devices can use the timestamp of a high-frequency sensor (such as a lidar, 10Hz) as a reference to linearly interpolate data from a low-frequency sensor (such as a camera, 5Hz) to generate a data frame aligned with the reference timestamp.

[0104] Electronic devices can downsample the target vehicle chassis operating status data (such as CAN bus 100Hz) and extract parameters such as speed and acceleration at the corresponding time according to the reference timestamp to ensure that they correspond one-to-one with driving scene image data, driving scene video data and driving scene point cloud data.

[0105] Then, electronic devices can use sensor extrinsic parameter calibration to map data from different coordinate systems such as cameras, lidar, and GPS to a unified coordinate system (usually the vehicle coordinate system: with the center of the rear axle of the vehicle as the origin, the forward direction as the X-axis, the left direction as the Y-axis, and the upward direction as the Z-axis).

[0106] Electronic devices can use a checkerboard calibration board to solve for the camera's intrinsic parameters (focal length, distortion coefficients) and extrinsic parameters (rotation matrix R, translation vector T) using calibration algorithms (such as OpenCV's calibrateCamera), transforming the image pixel coordinate system into the LiDAR coordinate system. Then, through vehicle body geometry measurement or motion calibration methods, the installation position of the LiDAR in the vehicle's coordinate system (e.g., X=0.5m, Y=0m, Z=1.8m) is determined, generating a coordinate transformation matrix. Finally, the GPS geodetic coordinate system (latitude and longitude) is transformed into the vehicle's local coordinate system using the INS tightly coupled algorithm.

[0107] For driving scene image data and driving scene video data, the electronic device transforms the data from the intrinsic parameter matrix to the camera coordinate system, then to the extrinsic parameter matrix, and finally to the vehicle coordinate system. For driving scene point cloud data, the electronic device directly transforms the data to the vehicle coordinate system via the extrinsic parameter matrix. For high-definition map data, the road coordinates (latitude and longitude) of the vector map are projected onto the vehicle's local coordinate system to achieve spatial alignment between the vehicle and the map.

[0108] Next, for the spatiotemporally aligned image and video frames, the electronic device divides the image into 16×16 local image blocks (e.g., a 224×224 image generates 196 image blocks). Each image block is flattened into a one-dimensional vector (16×16×3=768 dimensions), mapped to the model dimension (e.g., 768 dimensions) through an embedding layer, and a positional encoding is added (to record the spatial location information of the image block). Then, the electronic device captures the spatial dependencies between image blocks through a multi-head self-attention mechanism (e.g., recognizing the semantic relationship "traffic light is in front of the vehicle"), outputting CLStoken (global semantic features, 768 dimensions) or image block-level features (local semantic features). Finally, 768-dimensional semantic features of driving scene images and driving scene videos are obtained, including scene type (urban road / highway), target category (vehicle / pedestrian / traffic light), and target state (red light / vehicle changing lanes).

[0109] The electronic device performs denoising (outlier removal) and downsampling (FPS sampling to 1024 points) on the spatiotemporally aligned driving scene point cloud data. Then, the point cloud regions are divided according to the vehicle's coordinate system (e.g., forward 0-100m, lateral -50m to 50m). Center points of local regions are selected using FPS (e.g., 128 center points per region). For each center point, a local point set is created based on K-nearest neighbors or sphere radius. For each local point set, local geometric features (e.g., point cloud density, normal vectors) are extracted using PointNet and concatenated with the center point features. After multiple Set Abstraction layers, the final local features are globally pooled to generate 1024-dimensional driving scene point cloud semantic features. These features include obstacle location / size (e.g., a car 4.5m long and 1.8m wide is 30m ahead), road structure (lane lines / guardrail positions), and target movement trend (stationary / moving).

[0110] For the target vehicle chassis operating status data, the electronic device extracts chassis data within a continuous time window (e.g., 3 seconds) to form a time series (e.g., a speed sequence: [50, 52, 55, 53, 51] km / h). The sequence is standardized (value = (value - mean) / std) to eliminate dimensional differences. Then, time step position encoding is added to the time series (distinguishing between "second 1" and "second 2" speeds). A multi-head self-attention mechanism is used to capture temporal dependencies (e.g., recognizing acceleration behavior such as "speed increasing from 50 km / h to 55 km / h"). The temporal features output by the encoder are mean-pooled to generate 256-dimensional chassis operating status temporal features, including vehicle status (speeding / normal speed), motion behavior (rapid acceleration / sharp turning / smooth driving), and trajectory features (straight driving / turning).

[0111] For high-definition map data, electronic devices can parse the data into a graph structure, using road intersections as nodes and road segments as edges, adding attributes to nodes / edges (e.g., node attributes: traffic light / intersection type; edge attributes: speed limit 60km / h / number of lanes 3). The electronic device transforms node attributes (e.g., intersection type) into 128-dimensional node features through an embedding layer. It then aggregates neighboring node features using a Global Network (GCN) (e.g., the features of a given intersection are fused with attributes from three adjacent intersections) to update the node features. Finally, all node features are globally pooled to generate a 512-dimensional structured graph feature, including road topology (e.g., "current road segment connects 3 intersections"), traffic rules (speed limit / no turning / yield), and POI information (school / hospital locations).

[0112] Step S3032: Based on the target prompt words, extract the target semantic features from the semantic features corresponding to each modality data.

[0113] Specifically, the electronic device first inputs the target prompt word into a pre-trained text model (such as BERT or CLIP text encoder) to generate a semantic feature vector of the prompt word (such as 768-dimensional), which contains the core requirements of the task (such as "driving behavior score", "stability / safety", "speed / acceleration").

[0114] Then, the electronic device calculates the cosine similarity between the semantic features of each modality and the semantic features of the prompt words, and selects the feature dimensions or sub-vectors with similarity ≥ the threshold (e.g., 0.7) as the target semantic features.

[0115] The formula is as follows: .in, This represents the semantic feature vector of a certain modality. This is the semantic feature vector of the target prompt word. For example, Table 2 shows examples of target semantic feature extraction for different business types.

[0116] Table 2 Examples of target semantic feature extraction for different business types

[0117]

[0118] Step S3033: Perform feature fusion on the target semantic features to generate target fused features.

[0119] Specifically, step S3033 above may include the following steps:

[0120] Step b1: Select homologous modal semantic features from the target semantic features.

[0121] Specifically, electronic devices can filter out visual modal features by matching labels based on the fact that each feature vector in the target semantic feature set has a "modal label" (such as "feat_image_visual (image visual feature)", "feat_video_visual (video visual feature)", "feat_pointcloud_3d (3D point cloud feature)").

[0122] Visual modal features include "driving scene image semantic features (F)" img ")" and "Semantic features of driving scene videos (F)" video The homologous modal feature set of ").

[0123] Step b2 involves spatiotemporal attention aggregation of semantic features from the same modality to obtain unified semantic features.

[0124] Specifically, the homologous modal semantic features are driving scene image semantic features and driving scene video semantic features. Step b2 above may include the following steps:

[0125] Step b21: Calculate the temporal attention matrix corresponding to the semantic features of the driving scene video.

[0126] Specifically, let the semantic features of driving scene videos be... fvt is a 768-dimensional semantic feature vector output by a visual feature extraction network (such as ViT) for the t-th frame of the driving scene video. The sequence is standardized (mean 0, variance 1) to ensure that the features between frames are comparable.

[0127] Then, the electronic device can employ "autoregressive temporal attention" (based on TransformerEncoder) to define Query (current frame features), Key (features of all frames), and Value (features of all frames):

[0128] : ;

[0129] : ;

[0130] : ;

[0131] The formula for calculating the temporal attention matrix is: ;in, (Feature dimension) W is the normalization factor. q To query the projection matrix, W k W is the key projection matrix. v Let Q be the value projection matrix, where Q represents the query vector of the current frame, K represents the key vector of all frames, and V represents the value vector of all frames. Let T be the similarity score matrix, where T represents the matrix transpose, and T represents the calculation of the dot product of Q and K. The result measures the feature similarity between the current frame (Query) and historical / future frames (Key).

[0132] Electronic device output timing attention matrix ), temporal attention matrix element Attntemp [i,j] This represents the association weight between frame i and frame j (the larger the value, the stronger the dynamic association between the two frames).

[0133] Step b22: Based on the temporal attention matrix, calculate the temporal aggregation features corresponding to the semantic features of the driving scene video.

[0134] Specifically, the electronic device multiplies the temporal attention matrix (30×30 dimensional) with the value matrix (30×768 dimensional) to obtain a 30×768 dimensional weighted feature matrix, and then sums the results across the row dimension (frame dimension) to compress it into a 1×768 dimensional single-frame feature matrix. The calculation formula is as follows: .

[0135] Then, electronic devices Perform L2 normalization to ensure consistency with the semantic features of the driving scene image (F img The numerical ranges are consistent (both are [-1, 1]), resulting in 768-dimensional time-aggregated features. It contains the core dynamic semantics of the video sequence (such as "the car in front accelerates from a standstill to 30km / h" and "the traffic light changes from green to red").

[0136] Step b23: Obtain the spatial attention weight maps corresponding to the semantic features and temporal aggregation features of the driving scene image, respectively.

[0137] Specifically, electronic devices can utilize the spatial attention layer in the ViT (VisionTransformer) model to process semantic features of driving scene images (F... img For the corresponding image patch sequence, calculate the spatial attention weight of each image patch (reflecting whether the image patch is a core target region), and generate a spatial attention weight map corresponding to the semantic features of the driving scene image. (e.g., 224×224 pixels) The higher the weight value (closer to 1), the more likely the corresponding area is to be a core target (e.g., pedestrians, vehicles).

[0138] Then, electronic devices can aggregate temporal features based on the spatial attention layer in the ViT model. By using "dynamic spatial attention mapping" (projecting temporal dynamic semantics onto the spatial dimension), a spatial attention weight map corresponding to temporal aggregation features is generated. The weight of the core target area is also significantly higher than that of the background area (e.g., the area corresponding to the vehicle lane change trajectory has a high weight).

[0139] For example, the spatial attention weight map corresponding to the semantic features of driving scene images. The weight of the traffic light area is 0.95, the weight of the vehicle area is 0.9, and the weight of the background sky area is 0.1; the spatial attention weight map corresponding to the temporal aggregation feature. The weight of the area covered by the vehicle lane-changing trajectory is 0.92, the weight of the pedestrian movement path area is 0.88, and the weight of the background area is 0.08.

[0140] Step b24: Based on the spatial attention weight map corresponding to the semantic features of the driving scene image, determine the first core target region corresponding to the semantic features of the driving scene image.

[0141] Specifically, the electronic device can compare the spatial attention weight map corresponding to the semantic features of the driving scene image with a first preset weight threshold, retain the pixel regions in the semantic features of the driving scene image corresponding to the spatial attention weight map whose weights are greater than the first preset weight threshold, and merge adjacent core regions through connected component analysis (such as OpenCV's find Contours) to obtain multiple candidate regions.

[0142] Then, the electronic device can sort the candidate regions by area (number of pixels), retain the top 3 largest regions (such as vehicles, pedestrians, and traffic lights), and merge them into the first core target region R. img(e.g., “vehicle area (100,200)-(300,400) pixels + traffic light area (500,50)-(550,150) pixels”), and record its spatial coordinates (e.g., top left corner (x1,y1), bottom right corner (x2,y2)).

[0143] Step b25: Based on the spatial attention weight map corresponding to the temporal aggregation features, determine the second core target region corresponding to the temporal aggregation features.

[0144] Specifically, the electronic device can compare the spatial attention weight map corresponding to the temporal aggregation feature with a second preset weight threshold, retaining pixel regions in the temporal aggregation feature corresponding to the spatial attention weight map whose weights are greater than the second preset weight threshold. Adjacent core regions are then merged using connected component analysis (such as OpenCV's find Contours) to obtain multiple candidate regions. The electronic device can then sort the candidate regions by region area (number of pixels), retaining the top three largest regions (such as vehicles, pedestrians, and traffic lights), and merging them into a second core target region R. video And record its spatial coordinates.

[0145] Step b26: Calculate the spatial semantic mask overlap between the first core target region and the second core target region.

[0146] Specifically, the electronic device can be based on the first core target area R img Second core target region R video Generate a binary mask (core region pixels = 1, background = 0), denoted as M. img and M video (All dimensions are H×W).

[0147] Then, the electronic device uses the Intersection over Union (IoU) ratio to calculate the spatial semantic mask overlap between the first and second core target regions, using the formula: .

[0148] in, It is the area of ​​the intersection region of the two types of masks (number of overlapping pixels); It is the area of ​​the union region of the two types of masks.

[0149] The electronic device can compare the calculated spatial semantic mask overlap with a preset overlap threshold. If the spatial semantic mask overlap is greater than or equal to the preset overlap threshold, the first core target region and the second core target region are determined to be semantically consistent (the same target); if the spatial semantic mask overlap is less than the preset overlap threshold, the electronic device needs to perform the alignment operation in step b27 below.

[0150] Step b27: Align the first core target region and the second core target region based on the spatial semantic mask overlap.

[0151] Specifically, the electronic device extracts the first core target region R. img With the second core target area R video Key points (such as the region center, boundary corners), for example: R img Center: (x img ,y img )=(200,300); R video Center: (x video ,y video = (210, 305).

[0152] Electronic devices can calculate translation vectors based on the differences in key point coordinates. ), generate a 2D translation transformation matrix T. Then, for the second core target region R video Applying transformation matrix T to all pixel coordinates yields the aligned video kernel region R. video_align Ensure its compatibility with R img The IoU is greater than or equal to the preset overlap threshold (e.g., 0.85).

[0153] Step b28: The aligned first core target region and the second core target region are merged to generate a unified semantic feature.

[0154] Specifically, the electronic device can denote the aligned first core target region as The aligned second core target area is denoted as .

[0155] Then, the electronic device uses weighted fusion plus residual connection to fuse the aligned first and second core target regions, generating unified semantic features. The fusion formula is: .

[0156] in, The first core target area after alignment. For the aligned second core target area, For semantic features of driving scene images, For temporal aggregation features, λ is the spatial detail preservation coefficient (default 0.5, balancing the contribution of image and video features, which can be adjusted according to the actual situation); Res(·) is the residual connection, which preserves global semantic features and avoids information loss caused by local fusion. To unify semantic features.

[0157] Step b3: Select similar modal semantic features from the target semantic features that are similar to the unified semantic features.

[0158] Among them, similar modal semantic features and unified semantic features are both temporal semantic features.

[0159] Specifically, the electronic device can traverse all semantic features in the target semantic feature set and read the "modal attribute label" of each feature (such as "feat_unified_visual (unified visual feature)", "feat_pointcloud_3d (3D point cloud feature)", "feat_chassis_temporal (chassis temporal feature)"). Then, the electronic device can filter for the label "driving scene point cloud semantic feature" (denoted as F). pointcloud The core attributes of this feature are: a temporal point cloud sequence encoded by PointNet++ (e.g., 30 frames × 1024 dimensions), with each frame corresponding to the semantics of a 3D point cloud in the vehicle's coordinate system; including the target's 3D position, size, and motion trend (e.g., "the 3D coordinates of the vehicle in front change from (15,3,1.2) → (12,3,1.2), moving closer to the vehicle"). The electronic device will drive the scene point cloud semantic features (F... pointcloud (30 frames × 1024 dimensions), as similar modal temporal semantic features fused with unified semantic features.

[0160] Step b4: Perform cross-modal attention alignment on the unified semantic features and similar modal semantic features to obtain alternative semantic features.

[0161] Specifically, the similar modal semantic features are the point cloud semantic features of the driving scene. Step b4 above may include the following steps:

[0162] Step b41: Map the unified semantic features to the corresponding 3D space under the vehicle coordinates to obtain the first position code of the unified semantic features under the vehicle coordinates.

[0163] Specifically, electronic devices can utilize unified semantic features (F unified Extract the core target region (R) from the 768-dimensional (768-dimensional) region. img With R video_align The 2D pixel coordinates of the aligned region (e.g., the top left corner (x1, y1) and bottom right corner (x2, y2) of the vehicle region) are then used. Next, the 2D pixel coordinates are transformed into 3D spatial coordinates in the vehicle coordinate system using the "camera intrinsic matrix (K)" and "extrinsic matrix (T)". The formula is as follows: .

[0164] in, These are 2D pixel coordinates (including depth estimates, derived from the temporal dynamics of unified semantic features, such as "vehicle approaches → depth decreases"). The coordinates are 3D coordinates in the vehicle coordinate system (X: forward distance, Y: lateral offset, Z: height).

[0165] For example, an electronic device can employ sinusoidal position coding to map the 3D coordinates (X, Y, Z) of the core target to a high-dimensional space using "positional embedding," generating a 768-dimensional first position code PE1 consistent with the unified semantic feature dimension; the formula is:

[0166]

[0167] Among them, the numerical range of PE1 is normalized to [-1, 1], and F unified The numerical distribution is consistent.

[0168] Step b42: Map the semantic features of the driving scene point cloud to the corresponding 3D space under the vehicle coordinates to obtain the second position encoding of the semantic features of the driving scene point cloud under the vehicle coordinates.

[0169] Specifically, the semantic features F of the point cloud in the driving scene pointcloud The feature sequence (30×1024-dimensional) is encoded using PointNet++ for 30 frames of temporal point cloud data. Each frame's feature is bound to the 3D coordinates of the core target (e.g., the vehicle's center point (Xp, Yp, Zp)). Electronic devices can extract the 3D coordinates of the core target in each frame and take the temporal average as the global 3D position of the point cloud features (e.g., the average vehicle coordinates over 30 frames are (13.5, 3.2, 1.1) meters), ensuring consistency with the "single-frame aggregation characteristic" of the unified semantic features.

[0170] Then, the electronic device uses the exact same "sine position encoding" logic as in step b41 to map the 3D average coordinates (Xp, Yp, Zp) of the point cloud to a 768-dimensional second position code PE2. The encoding parameters of PE2 (such as the frequency coefficient 10000) are completely identical to those of PE1, ensuring semantic space consistency in the position encoding. The 768-dimensional second position code PE2 contains the 3D spatial position information of the core target in the point cloud's semantic features within the vehicle coordinate system, and is completely consistent with PE1 in terms of dimension, encoding logic, and numerical range.

[0171] Step b43: Calculate the semantic similarity between the unified semantic features and the similar modal semantic features.

[0172] Specifically, the similar modal semantic features (i.e., driving scene point cloud semantic features) are initially 1024-dimensional, and electronic devices can reduce their dimensionality to 768-dimensional using a "linear layer without activation function" (compared to the unified semantic feature F). unified (Consistent dimensions), formula: .in, For learnable weight matrix, For bias, The semantic features of similar modalities after dimensionality reduction are obtained. Pre-training with a "semantic loss function" ensures that 3D semantics are not lost after dimensionality reduction (the retention rate of target size and distance information is ≥95%).

[0173] For electronic devices, the cosine similarity formula can be used to calculate the semantic similarity between the dimensionality-reduced similar modal semantic features and the unified semantic features: ,in, For similar modal semantic features after dimensionality reduction, To unify semantic features.

[0174] Step b44: Calculate the positional similarity between the first positional code and the second positional code.

[0175] Specifically, the electronic device can extract the 3D coordinates (P1=(X1,Y1,Z1), P2=(X2,Y2,Z2)) corresponding to the first position code PE1 and the second position code PE2, and calculate the Euclidean distance: .

[0176] Electronic devices convert 3D distance into positional similarity within the [0,1] interval, using the following formula: Where α=0.5 (attenuation coefficient, optimized through the validation set to ensure similarity ≥0.8 when the distance is ≤0.5 meters and similarity ≤0.1 when the distance is ≥5 meters, which can be adjusted according to the actual situation).

[0177] Step b45: Based on semantic similarity and positional similarity, fuse the unified semantic features and similar modal semantic features to obtain alternative semantic features.

[0178] Specifically, step b45 above may include the following steps:

[0179] Step b451: Calculate the fusion weight information corresponding to the unified semantic features and similar modal semantic features based on semantic similarity and positional similarity.

[0180] Specifically, electronic devices can calculate the fusion weight information corresponding to unified semantic features and similar modal semantic features based on semantic similarity and positional similarity, respectively. The calculation formula is as follows: ,in, For semantic similarity, For positional similarity.

[0181] Step b452: Based on the fusion weight information, the unified semantic features and similar modal semantic features are fused to obtain backup semantic features.

[0182] Specifically, electronic devices can perform weighted fusion of unified semantic features and similar modal semantic features based on fusion weight information to obtain backup semantic features.

[0183] Step b5: Determine other semantic features in the target semantic features as heterogeneous modal semantic features corresponding to the backup semantic features.

[0184] Specifically, the electronic device can again traverse the target semantic feature set extracted in step S3032, excluding the two types of modal features (unified semantic feature F) that have already participated in the fusion. unified Point cloud semantic features F pointcloud The remaining features are labeled as "heterogeneous modal semantic features," which mainly include: chassis time-series data features (F...). chassis ): Temporal dynamics semantics (velocity, acceleration), no spatial semantics; map structure data features (F map ): Spatial rule semantics (speed limit, lane topology).

[0185] Step b6: Perform feature fusion between the backup semantic features and the heterogeneous modal semantic features to generate candidate semantic features.

[0186] Specifically, the heterogeneous modal semantic features include the structured graph features corresponding to the high-definition map and the time-series features of the chassis operating status. Step b6 above may include the following steps:

[0187] Step b61: Input the structured graph features and backup semantic features into the preset graph neural network attention module, and output the target-related map features.

[0188] Specifically, step b61 above may include the following steps:

[0189] Step b611: Transform the structured graph features corresponding to the high-definition map to generate map graph node features corresponding to the map nodes.

[0190] Specifically, the structured graph features of high-definition maps are typically stored in the form of a "graph," which is the core component. Nodes represent key road elements, such as intersections, traffic lights, lane endpoints, and Points of Interest (POIs) (schools / hospitals). Each node contains attribute information (e.g., "intersection type = cross," "traffic light status = red," "speed limit = 60km / h"). Edges represent the connections between nodes, such as "road segment from intersection A to intersection B" or "lane segment from lane endpoint 1 to lane endpoint 2." Edge attributes include "segment length," "number of lanes," and "no-turn rule," etc.

[0191] Electronic devices can use embedding encoding (such as "crossroads" → 128-dimensional vector) to encode...

[0192] Continuous attributes (such as speed limit and road segment length) are standardized (mean 0, variance 1) and the original values ​​are retained; rule attributes (such as no turning and yielding) are converted into binary features (no turning = 1, turning allowed = 0).

[0193] Then, the "discrete attribute embedding + continuous attribute standardized value + regular binary feature" of each node are concatenated to generate a graph node feature vector of uniform dimension (e.g., 512-dimensional), denoted as . Finally, the electronic device generates M feature vectors for each 512-dimensional map node (…). Each vector corresponds to a physical node (such as an intersection or traffic light) in a high-definition map, and contains the node's attributes, rules, location, and other core semantics.

[0194] Step b612: Input the map node features and backup semantic features into the preset graph neural network attention module.

[0195] Specifically, electronic devices will use "map node features (N)" map ")" and "alternate semantic features (F)" backup Input the preset GNN attention module and establish the connection between the two through the module's built-in spatial-semantic computation logic.

[0196] Step b613: The preset graph neural network attention module extracts the target geometric information corresponding to the target object from the backup semantic features.

[0197] The target geometry information includes the target object's location, size, and type.

[0198] Specifically, electronic devices can use a predefined "feature-geometric information mapping table" (established during the training phase) to obtain alternative semantic features F. backup The corresponding dimension is located in the middle.

[0199] The target geometric information includes: positional information dimensions (such as the 1st to 3rd dimensions): 3D coordinates in the vehicle coordinate system. (X: forward distance, Y: lateral offset, Z: height); Size information dimensions (e.g., dimensions 4-6): 3D dimensions of the target object. (Length × Width × Height, unit: meters); Category information dimension (e.g., dimensions 7-10): Target object category code (e.g., vehicle = 0010, pedestrian = 0001, traffic light = 0100).

[0200] Step b614: Calculate the Euclidean distance between each map node and each target object.

[0201] Specifically, electronic devices can be based on the 3D position of the target object. 3D location of map nodes Calculate the 3D Euclidean distance between the two: .

[0202] Then, the electronic device generates a distance matrix based on the Euclidean distance between each map node and each target object. It records the spatial distance between each map node and the target object.

[0203] Step b615: Based on the Euclidean distance, extract the target map nodes from each map node whose Euclidean distance to each target object is less than the preset Euclidean distance threshold.

[0204] Specifically, the electronic device can compare each Euclidean distance in the distance matrix with a preset Euclidean distance threshold. Then, the electronic device can extract target map nodes whose Euclidean distance to each target object is less than the preset Euclidean distance threshold from each map node, thus obtaining a set of target map nodes. .

[0205] Step b616: Determine the spatial weights of each target map node based on Euclidean distance.

[0206] Specifically, electronic devices can employ a "distance decay function" to calculate the spatial weight of each target map node based on Euclidean distance, ensuring that the weight monotonically decreases as the distance increases. The calculation formula is as follows: Where β=0.8 (attenuation coefficient, optimized through validation set to ensure that when distance=0, spatial weight≈1, and when Euclidean distance=preset Euclidean distance threshold, spatial weight≈0.45).

[0207] Step b617: Calculate the category semantic correlation between each target map node and each target object.

[0208] Specifically, the electronic device can extract category semantic sub-vectors (such as the first 64 dimensions of a 512-dimensional vector, corresponding to categories like "vehicles," "pedestrians," and "traffic lights") from the backup semantic features, denoted as S. obj .

[0209] The electronic device extracts functional semantic sub-vectors (such as the first 64 dimensions of the 512-dimensional vector, corresponding to functions such as "traffic light", "intersection", and "lane line") from the feature vector of each target map node nti, denoted as S. map,i .

[0210] Then, the electronic device can use the cosine similarity formula to calculate the semantic relevance between the two: Among them, Sim sem,i ∈[0,1], the higher the value, the stronger the semantic relevance.

[0211] Step b618: Determine the semantic weights of each target map node based on category semantic relevance.

[0212] Specifically, electronic devices can directly use semantic relevance Sim. sem,i As the original weight ( ).

[0213] Then, the electronic device can apply a non-linear activation (such as the Sigmoid function) to the original weights, strengthening the weights of highly relevant nodes and suppressing low-relevance nodes, to obtain the semantic weights corresponding to each target map node. The specific formula is as follows: Where γ=5 (activation coefficient, ensuring Simulation) sem,i W ≥0.7 sem,i ≥0.8, Sim sem,i Ws ≤0.3 em,i ≤0.3).

[0214] Step b619: Determine the node weights corresponding to each target map node based on spatial weights and semantic weights.

[0215] Specifically, electronic devices can employ a "weighted product + global normalization" strategy to determine the node weights corresponding to each target map node based on spatial and semantic weights, as shown in the following formula: .

[0216] in, Spatial weights, For semantic weights, For node weights,

[0217] Step b6110: Perform weighted fusion of each target map node based on node weights to output target-related map features.

[0218] Specifically, the electronic device can employ a "weighted summation of feature vectors" logic to perform weighted fusion of each target map node based on node weights, outputting target-related map features. The specific formula is as follows: .in, Let be the 512-dimensional feature vector of the i-th target map node; Let be the final node weight of the i-th target map node; This is a 512-dimensional feature vector, where the value of each dimension is a weighted sum of the corresponding dimension values ​​of all target map nodes (node ​​weights are...). ).

[0219] Step b62 involves fusing the timing features of the chassis operating status and the standby semantic features to obtain the target interactive chassis features.

[0220] Specifically, the electronic device uses a preset time-space fusion module to fuse the time-series features of the chassis operation status and the standby semantic features to obtain the target interactive chassis features.

[0221] The preset temporal-spatial fusion module is used to perform self-attention fusion processing after aligning the temporal features of the chassis operating status and the standby semantic features spatially and temporally. This preset temporal-spatial fusion module is the core neural network component for achieving spatiotemporal alignment of chassis temporal features with target dynamic features, and for risk perception and temporal fusion. It is not built independently, but rather constructed according to a data-driven modular design and phased training engineering logic.

[0222] Specifically, the pre-defined temporal-spatial fusion module consists of five interconnected sub-modules, acquired through two stages: offline pre-training and online inference. Step 1: Constructing the target dynamic feature extraction sub-module. The goal is to accurately extract the target's motion information from the backup semantic features. During the training phase, a feature-dynamic information mapping table is established based on a large-scale driving scenario dataset (containing different road conditions, vehicle speeds, and target types). This table defines the backup semantic features F. backup The dimensions of the relative position change (ΔX, ΔY) and target velocity (Vobj) are correspondingly indexed. A fully connected layer (FC) or a 1D convolutional layer is introduced to define the extraction operator. This operator has no complex nonlinear activation, ensuring linear preservation of positional information. Inputting 30 consecutive frames of prepared semantic features, the output is a target dynamic feature sequence with a dimension of 30×256.

[0223] Step 2: Construct a spatiotemporal alignment and feature pair generation submodule (geometric layer alignment). The aim is to solve the problem of heterogeneous data fusion due to "time asynchrony and inconsistent spatial coordinate systems," generating labeled training sample pairs. Design computational units based on interpolation or resampling. Using the timestamps of chassis features (t=1...30) as a benchmark, interpolate the target dynamic feature sequence to force a one-to-one correspondence at the frame level, constraining the time synchronization error to ≤0.05s. Introduce a rigid transformation matrix computational unit. Transform the target dynamic features uniformly to the vehicle's coordinate system and verify that the spatial deviation is ≤0.1m, ensuring that the target position change is synchronously reflected when the vehicle turns. Solidify the automatic classification rules into the module's internal logic. Input: Vego, aego, θego, ΔXobj, ΔYobj, Vobj. Output: One-Hot encoded labels (100 rear-end collision / 010 avoidance / 001 no interaction). Generate a 30-character sequence of chassis time-series features and target dynamic features, where each element is a triplet (f chassis (t),f obj_d(t),Label(t)).

[0224] Step 3: Train the risk feature similarity calculation submodule (risk layer modeling). The goal is to give the module the ability to "understand risk" and transform abstract interaction states into numerical scores.

[0225] Based on statistical big data, feature vectors f of three typical risk patterns are predefined. modeA ,f modeB ,f modeC The values ​​are then stored in the module parameters. Mode A (rear-end collision): Vego > Vobj, ΔXobj < 0, aego > 0. Mode B (avoidance): |ΔYobj| > 0.5m, |θego| > 5°. Mode C (no interaction): ΔXobj > 0, |ΔYobj| < 0.1m. Cosine similarity calculation logic cos(·,·) is embedded in the module. A maximum value operation max{…} is integrated to ensure a unique risk similarity score Simrisk(t) is output for each frame.

[0226] Step 4: Train the temporal self-attention weight calculation submodule (temporal layer focusing). The goal is to achieve a dynamic attention mechanism that "focuses on high-risk frames and weakens low-risk frames".

[0227] Based on the Transformer architecture, it integrates Query / Key / Value projection matrices (Wq, Wk, Wv). Input: Risk similarity vector Simrisk(t). Formula logic: .in, (Risk similarity is a scalar). Normalization factor; This represents the attention weight of frame t1 to frame t2, with higher weights between high-risk frames (e.g., if both frames are at risk of rear-end collision, the weight is approximately 0.8).

[0228] By using risk similarity as the benchmark for attention scores, the model is trained to give higher weights to highly similar frames (high risk). An integrated Softmax normalization layer is used to ensure that all weights sum to 1, thus guaranteeing gradient stability.

[0229] Step 5: Integrate the weighted fusion output submodule (feature layer generation). The goal is to aggregate the features of all frames according to their weights and output the final fused features with temporal context awareness.

[0230] Specifically, step b62 above may include the following steps:

[0231] Step b621: Extract the target dynamic features corresponding to the target object from the backup semantic features.

[0232] Among them, the target dynamic characteristics include the relative position change of the target object and the target velocity change.

[0233] Specifically, electronic devices can use a predefined "feature-dynamic information mapping table" (established during the training phase) to obtain backup semantic features F. backup The corresponding dimension is used for positioning. Among these, relative position change refers to the positional offset of the target object within consecutive frames in the vehicle coordinate system. , This reflects the target's direction of movement relative to the vehicle (e.g., approaching, moving away, or shifting laterally); target speed change: the target's instantaneous speed is:

[0234] Where Δt = 0.1 seconds is the frame interval) and the rate of change of velocity ( This reflects the motion state of the target object (such as uniform speed, acceleration, deceleration).

[0235] Electronic device outputs target dynamic feature sequence

[0236] (T=30 frames × 256 dimensions), each frame contains information on the relative position and velocity changes of the target object.

[0237] Step b622: Align the timing features of the chassis operation status with the target dynamic features corresponding to the target object in terms of time and space, and label the aligned timing features of the chassis operation status and the target dynamic features to generate multi-frame chassis timing feature-target dynamic feature pairs.

[0238] The annotation dimensions include the vehicle's dynamic state, the target's dynamic state, and the interaction type.

[0239] Specifically, electronic devices can use the timestamp of the timing characteristics of the chassis operation status as a reference (e.g., t=1,2,...,30 frames, with an interval of 0.1 seconds between each frame) to interpolate or downsample the target dynamic feature sequence, ensuring that each frame of chassis features corresponds to a unique frame of target dynamic features, with a time synchronization error ≤0.05 seconds.

[0240] Then, the electronic device can verify that the relative position change of the target dynamic feature sequence is based on the vehicle coordinate system and is consistent with the spatial reference of the chassis features (e.g., when the vehicle turns, the relative position change of the target needs to reflect the change of the vehicle's attitude synchronously), with a spatial deviation of ≤0.1 meters, thereby realizing the temporal and spatial alignment of the chassis operating state temporal features with the target dynamic features corresponding to the target.

[0241] The electronic device will label each aligned frame of data with three dimensions: vehicle dynamics, target dynamics, and interaction type, generating chassis time-series features-target dynamic features pairs. .

[0242] Wherein, Pai is a sequence of chassis temporal features and target dynamic features, which is composed of 30 frames of aligned feature pairs concatenated in chronological order; f chassis (t) represents the timing characteristics of the chassis operating state in frame t, including vehicle speed Vego(t), acceleration aego(t), and steering angle θego(t); f obj_d (t) represents the dynamic features of the target in frame t, including the relative position changes ΔXobj(t), ΔYobj(t), and the target velocity Vobj(t); Label(t) represents the interaction type label in frame t. The electronic device automatically classifies the interaction type based on the above states (e.g., "vehicle acceleration + target approaching → rear-end collision risk interaction", "vehicle turning + target lateral shift → avoidance interaction", "no significant relative motion → no interaction"), and uses One-Hot encoding to label the interaction type (e.g., rear-end collision risk interaction = 100, avoidance interaction = 010, no interaction = 001).

[0243] Step b623: Based on the interaction type, determine the similarity of risk features corresponding to the chassis temporal features and target dynamic features pairs in each frame.

[0244] Specifically, electronic devices can predefine three typical modes (based on big data statistics of driving scenarios) and store their feature vectors: Mode A (rear-end collision risk interaction, high risk): Vego>Vobj, ΔXobj<0 (objective approaches the vehicle), aego>0 (vehicle accelerates); Mode B (avoidance interaction, medium risk): |ΔYobj|>0.5 meters (objective shifts significantly to the side), |θego|>5° (vehicle turns); Mode C (no interaction, no risk): ΔXobj>0 (objective moves away), |ΔYobj|<0.1 meters, Vego≈Vobj.

[0245] For each frame, the electronic device calculates the cosine similarity between the feature vector of "vehicle dynamics state + target dynamics state" and the feature vector of the typical pattern, and takes the maximum value as the risk feature similarity of that frame.

[0246] ;

[0247] in, This represents the similarity of risk features in frame t. A higher value indicates that the current interaction is closer to a typical risk pattern (e.g., ...). →High risk →Low risk), where f pair (t) represents the chassis temporal features - target dynamic features pair in frame t, f modeA The feature vector f represents the pattern A (such as the interaction of rear-end collision risks). modeB The feature vector f represents pattern B (such as avoidance interaction).modeC Feature vector representing mode C (e.g., no interaction).

[0248] Step b624: Calculate the temporal self-attention weights based on the similarity of each risk feature to obtain the temporal self-attention weights corresponding to the chassis temporal feature-target dynamic feature pairs in each frame.

[0249] Specifically, the electronic device can take the chassis temporal feature - target dynamic feature pair as input, define Query=Key=Value=risk feature similarity vector of each frame chassis temporal feature - target dynamic feature pair, and calculate the temporal self-attention matrix:

[0250] ;in, (Risk similarity is a scalar). Normalization factor; This represents the attention weight of frame t1 to frame t2, with higher weights between high-risk frames (e.g., if both frames are at risk of rear-end collision, the weight is approximately 0.8).

[0251] Then, the electronic device can sum each row of the temporal self-attention matrix to obtain the temporal self-attention weights for each frame's feature pair: Next, electronic devices... Perform Softmax normalization to ensure that the sum of weights equals 1.

[0252] Finally, the electronic device obtains the corresponding temporal self-attention weights for each frame's chassis temporal feature-target dynamic feature pair. High-risk frames have a significantly higher weight than low-risk frames (e.g., rear-end collision risk frames have a weight of 0.15, and non-interactive frames have a weight of 0.01).

[0253] Step b625: Based on the temporal self-attention weights corresponding to the chassis temporal feature-target dynamic feature pairs in each frame, perform weighted fusion on the chassis temporal feature-target dynamic feature pairs in each frame to generate target interactive chassis features.

[0254] Specifically, the electronic device can perform weighted fusion of the chassis temporal features and target dynamic features of each frame based on the corresponding temporal self-attention weights of the chassis temporal features and target dynamic features of each frame, and generate target interactive chassis features.

[0255] The weighted fusion calculation formula is as follows: ;in, For temporal self-attention weights, For chassis time-series features - target dynamic features pairs, The target is an interactive chassis feature.

[0256] Step b63 involves cross-attention fusion of target-related map features, target-interactive chassis features, and backup semantic features to generate candidate semantic features.

[0257] Specifically, step b63 above may include the following steps:

[0258] Step b631: The alternative semantic features are determined as the first query vector.

[0259] Specifically, electronic devices can use semantic features F. backup ∈Rd (d is the feature dimension, such as 512 dimensions), directly define it as the first query vector: Q1=F backup Alternate semantic features are the core reference for multimodal fusion (such as containing key information like target object category and location). As a query, they ensure that subsequent attention calculations always revolve around the "target object semantics," avoiding deviation from the core task.

[0260] Step b632: The target-related map features and the target-interactive chassis features are concatenated to determine the first key vector.

[0261] Specifically, let the feature of the target-related map be F. map_target ∈Rd, the target interactive chassis feature is F chassis_target ∈Rd, the electronic device concatenates the target-related map features and the target-interactive chassis features to obtain the first key vector: The two d-dimensional features are concatenated into a 2d-dimensional vector (e.g., 512+512=1024 dimensions) to ensure that the Key contains the complete context of "environmental rules + vehicle interaction", providing a comprehensive matching basis for the Query.

[0262] Step b633: The target-related map features and the target-interactive chassis features are concatenated to determine the first value vector.

[0263] Specifically, let the feature of the target-related map be F. map_target ∈Rd, the target interactive chassis feature is F chassis_target ∈Rd, the electronic device concatenates the target-related map features and the target-interactive chassis features to obtain the first key vector: .

[0264] Step b634: Calculate the cross-attention weight matrix based on the first query vector and the first key vector.

[0265] Specifically, if the query dimension d and the key dimension 2d do not match, the electronic device can first project the query to the 2d dimension through a linear layer to obtain the final first query vector: .in, The projection weight matrix is... This is the projection bias vector.

[0266] The electronic device can calculate the cross-attention weight matrix based on the first query vector and the final first key vector, using the following formula: ,in, This is a dot product operation, which calculates the similarity between the Query and the Key across each dimension (the larger the value, the stronger the association). This is a normalization factor to avoid overflow of the dot product result due to excessively high key dimensions, thus ensuring stable Softmax calculation.

[0267] in, (e.g., 1024×1024 dimension), element Attncross [i,j] This represents the association weight between the i-th dimension of the query and the j-th dimension of the key.

[0268] Step b635: Multiply the cross-attention weight matrix with the first value vector to obtain the preliminary semantic features.

[0269] Specifically, the electronic device can multiply the cross-attention weight matrix with the first value vector to obtain preliminary semantic features, calculated as follows: For each dimension of V1, the electronic device uses attention weights to perform a weighted summation (dimensions with high weights contribute more, while dimensions with low weights are suppressed).

[0270] Among them, the output It retains "map + chassis semantics that are of interest to backup semantic features" (such as speed limit rules related to the target object and autonomous vehicle interaction risks), and removes irrelevant information (such as the topology of distant road sections and the chassis status without risks).

[0271] Step b636: The preliminary semantic features are fused with the backup semantic features to generate candidate semantic features.

[0272] Specifically, electronic devices can transmit preliminary semantic features The calculation formula is as follows: Dimensionality is reduced to d dimensions using a linear layer. ,in, For the dimension reduction weight matrix, This is the dimension reduction bias vector.

[0273] Then, the electronic device uses residual connections and layer normalization to fuse the dimensionality-reduced preliminary semantic features with the backup semantic features to generate candidate semantic features, as shown in the formula: .

[0274] in, These are the initial semantic features after dimensionality reduction. Residual connections are used as backup semantic features. It retains the core information of the backup semantic features (such as the object category) while incorporating the contextual semantics of the map and chassis (such as speed limit rules); layer normalization stabilizes the feature distribution, avoids gradient fluctuations, and improves the convergence of subsequent tasks; the final output is... These are candidate semantic features, which combine multimodal context and core target semantics.

[0275] For example, such as Figure 4 The diagram shown is a design diagram of a multi-layer, multi-source feature adapter. The adaptation process follows an alignment strategy of "from near to far, from structure to semantics":

[0276] (1) Homologous modality alignment: First, homologous modalities with similar spatial structures and information densities are aligned, such as image and video frame features. Inter-frame information aggregation and feature consistency are achieved through a spatiotemporal attention mechanism. The spatiotemporal attention mechanism can be directly aggregated using existing methods.

[0277] (2) Approximate Modal Alignment: Subsequently, image features and point cloud features are aligned. Both contain environmental geometric information, but their data sampling methods and dimensional structures differ. Through 3D-2D mutual mapping and cross-modal attention, the geometric positions and object boundaries are made consistent. The BEV method (Bevformer, etc.) projects multi-view image information into 3D space, where each target has its own 3D position. Since point cloud information is itself 3D information, alignment is achieved in 3D space. At the same time, through coordinate system transformation (based on camera and laser calibration parameters), the information can be mapped back to 2D space and aligned with elements on the image. Mutual mapping and delivery can further improve information accuracy.

[0278] (3) Heterogeneous modality fusion: After the geometric features are stabilized, modal features that differ significantly from the spatial structure are gradually introduced, including road topology information of the map (structured graph features) and continuous temporal dynamic features of the chassis (temporal vectors). These are accessed into the global feature space through graph neural network attention module (GNN) and temporal-spatial fusion module (TemporalSelf-Attention), respectively. The implementation of this part is relatively straightforward; after feature extraction, it can be directly fused through a cross-attention mechanism.

[0279] (4) High-level semantic alignment: Finally, the fused multimodal environmental features are aligned with the high-level semantic features (Prompt encoded representation) across the semantic space. Through the task-conditioned attention mechanism, the multi-source features are mapped to a common representation space consistent with the task semantics, thereby supporting multi-task reasoning. Simply put, the task-conditioned attention scheme is implemented using a multi-head cross-attention mechanism. The number of heads is consistent with the input data, and each head can physically represent an input, thus enabling targeted learning.

[0280] The core technical advantages of this multi-layer, multi-source feature adaptation strategy are: (1) it adopts a progressive alignment process, which effectively reduces information conflicts caused by direct full-modal fusion; (2) it designs a differentiated adaptation mechanism for geometric, temporal, and topological differences between different modalities, which improves the accuracy and robustness of feature alignment; and (3) it introduces high-level semantic guidance during the fusion process, making the final feature expression more in line with the specific task requirements. After completing the multi-layer alignment, the fused features and the corresponding task prompt are input into the large language model, and the LLM performs cross-modal reasoning, generates analysis reports, and outputs task results (including label list, score, rule judgment, matching degree value, quality score, etc.).

[0281] Step b7: Fuse the candidate semantic features with the semantic features of the prompt word corresponding to the target prompt word to generate the target fused feature.

[0282] Specifically, step b7 above may include the following steps:

[0283] Step b71: Identify the semantic features of the prompt words corresponding to the target prompt words, and determine the number of tasks corresponding to the semantic features of the prompt words.

[0284] Specifically, electronic devices can use preset "semantic feature-task mapping rules" or lightweight classifiers to analyze the semantic features F of prompt words. prompt Extract the number of tasks N task .

[0285] Example 1: Prompt "Generate road scene object detection labels" → N task =1 (Single detection task); Example 2: Prompt "Detect vehicles and divide lane lines" → N task =2 (Dual task); Example 3: Prompt words "Detect target, assess risk, plan path" → N task =3 (Multitasking).

[0286] Step b72: Determine the number of attention heads based on the number of tasks.

[0287] Optionally, the electronic device can determine the number of tasks as the number of attention heads.

[0288] Optionally, the electronic device can determine the number of attention heads based on the mapping relationship between the number of tasks and the number of attention heads. For example, for a single task, one task requires two heads to capture "global correlation" and "local details" respectively; for a dual task, each task is assigned two heads to take into account both intra-task correlation and inter-task correlation; for three or more tasks (multi-task), more heads are needed to cover the core dimensions of each task.

[0289] Step b73: Concatenate the candidate semantic features with the prompt word semantic features to generate concatenated semantic features.

[0290] Specifically, electronic devices can concatenate candidate semantic features with prompt word semantic features to generate concatenated semantic features.

[0291] For example, the alternative semantic features are The semantic features of the prompt words are After piecing them together, we get: The two d-dimensional features are concatenated into a 2d-dimensional vector (e.g., 512+512=1024 dimensions) to ensure that they contain complete information of "data semantics + task semantics", providing comprehensive input for subsequent attention calculation.

[0292] Step b74: Generate the total query vector, total key vector, and total value vector based on the concatenated semantic features.

[0293] Specifically, electronic devices can generate the total query vector through three independent linear layers: Total key vector (Key) Total Value Vector .

[0294] in, As a learnable weight matrix, the 2D concatenated features are mapped to... Total vector of dimension (e.g., H=4, d) k When =128, ). As the bias vector, adjust the vector distribution.

[0295] Step b75: The total query vector, total key vector, and total value vector are split into subquery vector, subkey vector, and subvalue vector respectively according to the number of attention heads.

[0296] Each attention head corresponds to a subquery vector, a subkey vector, and a subvalue vector.

[0297] Specifically, the electronic device splits the total query vector, total key vector, and total value vector into H sub-vectors based on the number of attention heads H: ; ; .

[0298] Each subvector has a dimension of d. k =d / H, total dimension preserved (e.g., a 512-dimensional total vector is split into four 128-dimensional sub-vectors); each attention head h corresponds to a set of (Q) vectors. h ,K h V h This ensures that the head and sub-vectors are bound one-to-one.

[0299] Step b76: For each attention head, calculate the attention weight corresponding to the attention head based on the subquery vector, subkey vector, and subvalue vector corresponding to the attention head.

[0300] Specifically, for each attention head, the attention weight corresponding to the attention head is calculated based on the subquery vector, subkey vector, and subvalue vector corresponding to the attention head.

[0301] For example, taking the h-th attention head, the attention weight corresponding to the h-th attention head is calculated as follows: .in, The dot product operation between the sub-Query and the sub-Key is used to calculate the similarity between dimensions. This is a normalization factor to prevent overflow of the dot product result due to excessive dimensionality. Let h be the attention weight matrix for the h-th attention head, with element Attnh. [i,j] This represents the association weight between the i-th dimension of the sub-Query and the j-th dimension of the sub-Key (the larger the value, the stronger the association).

[0302] The weights of different heads focus on different task dimensions (e.g., head 1 focuses on "detection task - target location association", head 2 focuses on "scoring task - speed association").

[0303] Step b77: Based on the attention weights corresponding to the attention heads, calculate the sub-value vectors corresponding to the attention heads and perform a weighted summation to obtain the attention head output corresponding to the attention heads.

[0304] Specifically, for each attention head, the electronic device can calculate the weighted sum of the sub-value vectors corresponding to the attention head to obtain the attention head output corresponding to the attention head.

[0305] For example, for the h-th attention head, the corresponding attention head output is calculated as follows: By using attention weights Attn hPair ValueV h The weighted summation of each dimension highlights the contribution of highly correlated dimensions (such as the detection task head which strengthens the information of the "target location dimension") and suppresses low-correlation dimensions.

[0306] Step b78: Concatenate the outputs of each attention head to obtain the target fusion feature.

[0307] Specifically, the electronic device concatenates the outputs of each attention head to form the target fusion feature: .

[0308] The output of each head is d. k The total dimensions after splicing are: (For example, four 128-dimensional head outputs are concatenated to form a 512-dimensional structure), consistent with the original feature dimensions, facilitating use in subsequent tasks. Among these, the target fusion features... It simultaneously includes "data semantics" containing candidate features and "task semantics" containing prompt word features, and the correlation information of different task dimensions is accurately captured by each attention point.

[0309] Step S3034: Input the target prompt word and the target fusion feature into the large language model, and output the task result corresponding to the target prompt word.

[0310] Specifically, electronic devices can convert target prompts (such as "analyze the safety of the current driving scenario and score it") into a token sequence of a large language model, denoted as... (n is the token length, t) i (This is the index in the model dictionary corresponding to the i-th token).

[0311] Target fusion features (e.g., 512-dimensional) is a numerical vector, and electronic devices can fuse target features through a "feature embedding layer". Transform into target fusion embedding features of the same dimension as the token embedding of a large language model : .in For feature embedding weight matrix, For feature embedding bias vector, For the hidden layer dimension of a large language model (e.g., 768 dimensions), ensure that the target fusion feature embedding and the token embedding dimension are consistent.

[0312] Then, the electronic device obtains the prompt word embedding sequence of the prompt word token, represented as: ( (The embedding of the i-th token).

[0313] Then, the electronic device fuses the target with embedded features. As a "special token embedding," it is inserted at a specified position (such as the beginning or end) into the prompt word embedding sequence to form the input embedding sequence. For example, Inserting target fusion feature embeddings at specified positions in the prompt word embedding sequence allows large language models to prioritize the core semantics of the scene data and then combine them with prompt words to understand task requirements.

[0314] The electronic device adds positional encoding to the input embedded sequence to generate the final input sequence, as shown in the following formula: Then, the electronic device generates a Query, Key, and Value vector based on the final input sequence: ; ; .in, For the attention projection weight matrix, This is the input vector for the attention mechanism.

[0315] After the electronic device is split into attention heads H, the attention weight of each head is calculated independently. For example, the attention weight of the h-th attention head is calculated as follows:

[0316]

[0317] in, This is a single-head dimension. The large language model automatically strengthens the weights of the corresponding dimensions in the fused features based on the task keywords (such as "tag," "score," and "rule determination") in the target prompt. For example, if the target prompt requires "target tag," the attention head focuses on the "target category" and "location" dimensions in the fused features; if it requires "safety score," it focuses on the "vehicle speed," "obstacle distance," and "speed limit rule" dimensions. Finally, the electronic device generates an attention output containing cross-modal correlation information. .

[0318] Next, the attention output is processed by a feedforward network (FFN) for non-linear feature enhancement. Then, by stabilizing the distribution through layer normalization, the normalized context is obtained: .

[0319] The normalized context representation is input into the large language model's generation layer and converted into a dictionary-dimensional probability distribution: ; .in, P(t) is the vector at the last position of the sequence, corresponding to the ending semantics of the target prompt word; P(t) is the generation probability of each token.

[0320] The electronic device selects the token with the highest probability (such as "目") as the first output. Then, the embedding of this token is added to the input sequence, and the inference process in step 3 is repeated to generate the next token (such as "标" → "列" → "表" → ":"); until the end symbol is generated ( <eos>(or reaches the preset length, ultimately yielding the original token sequence:) Example output sequence: ["target","column","table",":","vehicle","vehicle","walk","person",";","evaluation","score","8","5","score",";","rules","judgment","determined",":","compliance","rules"].

[0321] Finally, the electronic device maps the output token sequence into natural language text. For example, "Target list: vehicles, pedestrians; score: 85 points; rule judgment: compliant; matching degree: 92%; quality score: 90 points".

[0322] For example, such as Figure 5 The diagram shown is a schematic representation of the overall framework provided in an embodiment of this application. Figure 3 As shown, electronic devices can acquire information in real time from multiple heterogeneous data sources to ensure comprehensive perception of the environment and vehicle status, including: Image: visual data from cameras; Point Cloud: 3D spatial structure data from LiDAR; Chassis Data: vehicle status information (such as vehicle speed, steering angle, braking status, etc.); Map Data: high-precision map information (such as lane lines, speed limits, road topology, etc.).

[0323] For different types of data, dedicated encoders are used to transform them into high-dimensional feature vectors that can be understood by machines: Image / video encoder: extracts visual features such as targets, scenes, and semantics from images; Point cloud encoder: processes 3D point cloud data and extracts geometric features such as object shape, position, and size; Chassis data encoder: maps the dynamic numerical data of vehicles into feature vectors; Map data encoder: parses the spatial relationships and topological structure in maps.

[0324] Then, feature vectors from images, point clouds, chassis, maps, and other sources with different dimensions and distributions are mapped and aligned in a unified space, and then deeply fused. This solves the problem of "different corpora" in multi-source data, allowing the system to understand the logical relationships between different data (such as the relationship between "seeing a red light" and "vehicle slowing down").

[0325] The Knowledge Base injects the underlying logic of the system's operation, including: prior driving knowledge (such as traffic rules and driving etiquette); common sense and world knowledge (such as physical laws and social common sense); and multi-task prompts (explicitly informing the model of the specific tasks to be performed, including: intelligent driving labeling system; driving behavior understanding and scoring; multi-source information matching evaluation; traffic scene understanding; and data quality evaluation scoring). The text encoder converts the natural language prompts into vectors.

[0326] Large Language Models (LLMs) receive fused multimodal feature vectors, text prompts, and external knowledge bases, and perform comprehensive logical reasoning and semantic generation. For responses to different tasks, the model outputs structured natural language answers, evaluation reports, or decision suggestions based on the task specified by the front end (such as scene understanding or behavior scoring).

[0327] For example, such as Figure 6 The diagram shown is a design diagram of the semantic framework for intelligent driving multi-task prompts provided in this application embodiment. The core semantic foundation layer (intelligent driving prompt semantic ontology) is the logical core of the entire system, defining the semantic standards for intelligent driving tasks and ensuring the accuracy of model understanding. It includes: Concept: Defines the basic elements of perception. Content: Perceived targets (e.g., vehicles, pedestrians, traffic lights), spatial attributes (e.g., lane line position, object distance), attribute features (e.g., color, model). Relation: Defines the associations between elements. Content: Relationships between targets (e.g., rear-end collision, pedestrian on a zebra crossing), relationships between targets and rules (e.g., red light prohibits passage). Behavior: Defines executable driving actions. Content: Lane changing, overtaking, detouring, and strategic maneuvering (referring to game strategies in conflict scenarios). Decision: Defines the final driving intention. Content: Acceleration, deceleration, braking, avoidance, and stopping. This layer provides strict ontological constraints for the LLM, preventing the model from outputting non-compliant or logically confused driving commands.

[0328] The multi-task prompt word and domain task layer are responsible for transforming specific business requirements into prompts that the model can understand. The intelligent driving multi-task prompt word semantic framework directly specifies the specific functions that the LLM needs to complete. These include: intelligent driving label system construction, driving behavior understanding and scoring, multi-source information matching degree evaluation, traffic scene and traffic rule understanding, and data quality assessment. The intelligent driving domain task layer (expandable) is an open task pool that not only includes basic perception and decision-making but also extends to advanced tasks such as scene label construction and abnormal behavior detection.

[0329] The multimodal data processing and fusion layer is a crucial step in the system's handling of the external environment and vehicle status. The text encoder transforms the natural language prompts from the second step into machine-processable vector features. The cross-modal progressive feature alignment network (PCAN) is the heart of the architecture. It processes multi-source data from the physical world (images, point clouds, chassis, maps, etc., corresponding to the input of your previous image) and aligns them with concepts and relations from semantic ontology. It bridges the gap between "raw sensor data" and "abstract semantic rules," enabling the LLM to understand the "dialect" of the hardware.

[0330] The Large Language Model (LLM) inference layer (decision center) acts as the system's "brain." Inputs include: aligned multimodal features (from PCAN): the real-time state of the environment; task cue word vectors (from the text encoder): explicit target instructions; and semantic ontology constraints: underlying logical rules. The LLM combines these inputs for deep semantic understanding and logical reasoning. The output layer (task output and feedback) provides different responses for different domain tasks: for perception tasks, it outputs a semantic description of the scene; for decision-making tasks, it outputs specific driving action instructions (e.g., "decelerate and avoid"); and for evaluation tasks, it outputs a quantified score report.

[0331] This embodiment provides a method for using large models based on multimodal data. It extracts semantic features from each modality and mines the core semantics of the multimodal data (such as target category, speed, and road rules), providing a high-quality semantic foundation for subsequent feature fusion. Based on prompt words, it filters target semantic features strongly related to the task (such as target location features for detection tasks), reducing irrelevant semantic interference and improving fusion efficiency. It aggregates semantic features from the same source (such as image + video features from the same visual modality), strengthening the consistency and complementarity of information from the same source. Based on spatiotemporal attention aggregation, it integrates the spatiotemporal correlations of the same modality (such as video temporal dynamics + image spatial details), forming a core semantic representation that is both complete and coherent. It filters semantic features of similar modalities (temporal categories), focusing on temporally related modalities (such as chassis temporal data + video temporal features), ensuring semantic correlation across modalities. Cross-modal attention alignment yields backup semantic features, eliminating semantic biases in similar modalities, generating backup features that combine the advantages of multiple modalities, and improving the robustness of semantic expression. Identify heterogeneous modal semantic features and identify heterogeneous modalities that complement the backup features (such as map rule features) to provide differentiated information for comprehensive fusion. Fusion generates candidate semantic features, integrating backup and heterogeneous features to achieve full coverage of "core semantics + complementary information," enriching the semantic dimensions of the features. Fusion of prompt word semantic features generates target fusion features, binding "data semantics" and "task semantics," enabling features to be both scenario-supported and task-oriented, improving the accuracy of large-scale model inference.

[0332] Finally, the target prompt words and target fusion features are input into the large language model, and the task results corresponding to the target prompt words are output. With the help of the cross-modal reasoning capabilities of the large model, standardized results are directly output without the need for independent modeling, thus reducing development costs; at the same time, the interpretability of the results and the credibility of the decision are improved.

[0333] This embodiment also provides a large model utilization device based on multimodal data, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0334] This embodiment provides a device for using large models based on multimodal data, such as... Figure 7 As shown, it includes:

[0335] The acquisition module 401 is used to acquire multimodal data of the target vehicle based on the target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data.

[0336] Module 402 is used to determine the current business type based on actual business needs; generate a prompt framework corresponding to the target prompt word based on the current business type; the prompt framework includes role positioning, rules, and constraints; extract target modality data from multimodal data based on the prompt framework; and input the target modality data into the prompt framework to generate the target prompt word.

[0337] The output module 403 is used to perform semantic recognition on multimodal data, extract semantic features corresponding to each modality of data; extract target semantic features from the semantic features corresponding to each modality of data based on the target prompt word; perform feature fusion on the target semantic features to generate target fusion features; input the target prompt word and the target fusion features into the large language model, and output the task result corresponding to the target prompt word.

[0338] The large model application apparatus based on multimodal data provided in this embodiment of the invention can execute the large model application method based on multimodal data provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0339] This application provides a target vehicle, including a vehicle body and electronic devices. The electronic devices are used to perform the large model usage method based on multimodal data according to any of the above embodiments. Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0340] The following is a detailed reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 01, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 02 or a program loaded from a memory 08 into a random access memory (RAM) 03. The RAM 03 also stores various programs and data required for the operation of the electronic device. The processor 01, ROM 02, and RAM 03 are interconnected via a bus 04. An input / output (I / O) interface 05 is also connected to the bus 04.

[0341] Typically, the following devices can be connected to I / O interface 05: input devices 06 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 07 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 08 including, for example, magnetic tapes, hard disks, etc.; and communication devices 09. Communication device 09 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0342] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 09, or installed from memory 08, or installed from ROM 02. When the computer program is executed by processor 01, it performs the functions defined in the large model usage method based on multimodal data of the embodiments of the present invention.

[0343] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0344] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded via a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the large model usage method based on multimodal data shown in the above embodiments is implemented.

[0345] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0346] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.< / eos>

Claims

1. A method for using large models based on multimodal data, characterized in that, The method includes: Acquire multimodal data of the target vehicle based on the target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data. Determine the current business type based on actual business needs; Based on the current business type, a prompt framework corresponding to the target prompt word is generated; the prompt framework includes role positioning, rules, and constraints. Based on the aforementioned prompting framework, target modality data is extracted from the multimodal data; The target modality data is input into the prompting frame to generate target prompt words; Semantic recognition is performed on the multimodal data to extract the semantic features corresponding to each modality; Based on the target prompt words, target semantic features are extracted from the semantic features corresponding to each modality of data; The target semantic features are fused to generate target fused features; The target prompt word and the target fusion feature are input into the large language model, and the task result corresponding to the target prompt word is output. The step of fusing the target semantic features to generate target fused features includes: Select homologous modal semantic features from the target semantic features; Spatiotemporal attention aggregation is performed on the semantic features of the homologous modalities to obtain unified semantic features; From the target semantic features, similar modal semantic features that are similar to the unified semantic features are selected; the similar modal semantic features and the unified semantic features are both temporal semantic features; Cross-modal attention alignment is performed on the unified semantic features and the similar modal semantic features to obtain alternative semantic features; Other semantic features in the target semantic features are identified as heterogeneous modal semantic features corresponding to the backup semantic features; The backup semantic features are fused with the heterogeneous modality semantic features to generate candidate semantic features; The candidate semantic features are fused with the semantic features of the prompt word corresponding to the target prompt word to generate the target fused feature.

2. The method according to claim 1, characterized in that, The step of inputting the target modality data into the prompting frame to generate target prompt words includes: The target modal data is converted into structured descriptive features; The structured description features are input into the prompting framework to generate initial prompt words; Based on the current business type, determine the task constraints and output requirements corresponding to the initial prompt word; Based on the task constraints and output requirements, the initial prompt words are completed to generate the target prompt words.

3. The method according to claim 1, characterized in that, The homologous modal semantic features are driving scene image semantic features and driving scene video semantic features. The spatiotemporal attention aggregation of the homologous modal semantic features to obtain unified semantic features includes: Calculate the temporal attention matrix corresponding to the semantic features of the driving scene video; Based on the temporal attention matrix, calculate the temporal aggregation features corresponding to the semantic features of the driving scene video; Obtain the spatial attention weight maps corresponding to the semantic features and temporal aggregation features of the driving scene image, respectively; Based on the spatial attention weight map corresponding to the semantic features of the driving scene image, the first core target region corresponding to the semantic features of the driving scene image is determined; Based on the spatial attention weight map corresponding to the temporal aggregation feature, the second core target region corresponding to the temporal aggregation feature is determined. Calculate the spatial semantic mask overlap between the first core target region and the second core target region; Based on the spatial semantic mask overlap, the first core target region and the second core target region are aligned; The aligned first and second core target regions are merged to generate the unified semantic feature.

4. The method according to claim 1, characterized in that, The similar modal semantic features are point cloud semantic features of driving scenes. Cross-modal attention alignment is performed on the unified semantic features and the similar modal semantic features to obtain backup semantic features, including: The unified semantic features are mapped to the corresponding 3D space under the vehicle coordinate system to obtain the first position code of the unified semantic features under the vehicle coordinate system. The semantic features of the driving scene point cloud are mapped to the corresponding 3D space under the vehicle coordinates to obtain the second position encoding of the semantic features of the driving scene point cloud under the vehicle coordinates. Calculate the semantic similarity between the unified semantic features and the similar modal semantic features; Calculate the positional similarity between the first positional code and the second positional code; Based on the semantic similarity and the positional similarity, the unified semantic features and the similar modal semantic features are fused to obtain the backup semantic features.

5. The method according to claim 4, characterized in that, The step of fusing the unified semantic features and the similar modality semantic features based on the semantic similarity and the positional similarity to obtain the backup semantic features includes: Based on the semantic similarity and the positional similarity, calculate the fusion weight information corresponding to the unified semantic feature and the similar modality semantic feature, respectively; Based on the fusion weight information, the unified semantic feature and the similar modality semantic feature are fused to obtain the backup semantic feature.

6. The method according to claim 1, characterized in that, The heterogeneous modal semantic features include structured graph features corresponding to the high-definition map and time-series features of chassis operating status. The process involves fusing the backup semantic features with the heterogeneous modal semantic features. Generate candidate semantic features, including: The structured graph features and the backup semantic features are input into a preset graph neural network attention module, which outputs target-related map features. The timing features of the chassis operating status and the standby semantic features are fused to obtain the target interactive chassis features; The target-related map features, the target-interactive chassis features, and the backup semantic features are cross-attention fusions to generate the candidate semantic features.

7. The method according to claim 6, characterized in that, The step of inputting the structured graph features and the backup semantic features into the preset graph neural network attention module and outputting target-related map features includes: The structured graph features corresponding to the high-definition map are transformed to generate map graph node features corresponding to the map nodes; The map node features and the backup semantic features are input into the preset graph neural network attention module; The preset graph neural network attention module extracts the target geometric information corresponding to the target object from the backup semantic features; the target geometric information includes the position, size, and type of the target object. Calculate the Euclidean distance between each of the map nodes and each of the target objects; Based on the Euclidean distance, target map nodes whose Euclidean distance to each target object is less than a preset Euclidean distance threshold are extracted from each of the map nodes. Based on the Euclidean distance, determine the spatial weight corresponding to each of the target map nodes; Calculate the category semantic correlation between each of the target map nodes and each of the target objects; Based on the semantic relevance of the categories, determine the semantic weight corresponding to each of the target map nodes; Based on the spatial weights and the semantic weights, the node weights corresponding to each of the target map nodes are determined; The target map nodes are weighted and fused based on the node weights to output the target-related map features.

8. The method according to claim 6, characterized in that, The process of fusing the timing features of the chassis operating status and the standby semantic features to obtain the target interactive chassis features includes: Extract the target dynamic features corresponding to the target object from the backup semantic features; the target dynamic features include the relative position change and target velocity change of the target object. The chassis operating state temporal features are aligned temporally and spatially with the target dynamic features corresponding to the target object. The aligned chassis operating state temporal features and target dynamic features are labeled to generate multi-frame chassis temporal feature-target dynamic feature pairs. The labeling dimensions include vehicle dynamics state, target dynamic state, and interaction type. Based on the interaction type, determine the risk feature similarity of the chassis temporal feature-target dynamic feature pair in each frame; Based on the similarity of each risk feature, the temporal self-attention weight is calculated to obtain the temporal self-attention weight corresponding to the chassis temporal feature-target dynamic feature pair in each frame. Based on the temporal self-attention weights corresponding to the chassis temporal features-target dynamic features pairs in each frame, the chassis temporal features-target dynamic features pairs in each frame are weighted and fused to generate the target interactive chassis features.

9. The method according to claim 6, characterized in that, The process of performing cross-attention fusion on the target-related map features, the target-interactive chassis features, and the backup semantic features to generate the candidate semantic features includes: The alternative semantic features are determined as the first query vector; The target-related map features and the target-interactive chassis features are concatenated to determine the first key vector; The target-related map features and the target-interactive chassis features are concatenated to determine the first value vector; Calculate the cross-attention weight matrix based on the first query vector and the first key vector; Multiplying the cross-attention weight matrix by the first value vector yields preliminary semantic features; The preliminary semantic features are fused with the backup semantic features to generate the candidate semantic features.

10. The method according to claim 1, characterized in that, The step of fusing the candidate semantic features with the semantic features of the prompt word corresponding to the target prompt word to generate the target fused feature includes: Identify the semantic features of the prompt words corresponding to the target prompt words, and determine the number of tasks corresponding to the semantic features of the prompt words; Based on the number of tasks, determine the number of attention heads; The candidate semantic features are concatenated with the prompt word semantic features to generate concatenated semantic features; Generate a total query vector, a total key vector, and a total value vector based on the concatenated semantic features; The total query vector, total key vector, and total value vector are respectively divided into subquery vectors, subkey vectors, and subvalue vectors according to the number of attention heads; wherein each attention head corresponds to one subquery vector, one subkey vector, and one subvalue vector. For each attention head, the attention weight corresponding to the attention head is calculated based on the sub-query vector, the sub-key vector, and the sub-value vector corresponding to the attention head. Based on the attention weights corresponding to the attention heads, the sub-value vectors corresponding to the attention heads are calculated and weighted summed to obtain the attention head outputs corresponding to the attention heads. The outputs of each attention head are concatenated to obtain the target fusion feature.

11. A device for using large models based on multimodal data, characterized in that, The device includes: The acquisition module is used to acquire multimodal data of the target vehicle based on the target driving scenario; the multimodal data includes at least two of the following: driving scenario image data, driving scenario video data, driving scenario point cloud data, target vehicle chassis operating status data, and high-definition map data. A construction module is used to determine the current business type based on actual business needs; generate a prompt framework corresponding to the target prompt word based on the current business type; the prompt framework includes role positioning, rules, and constraints; extract target modality data from the multimodal data based on the prompt framework; and input the target modality data into the prompt framework to generate the target prompt word. The output module is used to perform semantic recognition on the multimodal data and extract semantic features corresponding to each modality; based on the target prompt word, extract target semantic features from the semantic features corresponding to each modality data; perform feature fusion on the target semantic features to generate target fusion features; input the target prompt word and the target fusion features into a large language model and output the task result corresponding to the target prompt word; the feature fusion on the target semantic features to generate target fusion features includes: filtering out homologous modal semantic features from the target semantic features; performing spatiotemporal attention aggregation on the homologous modal semantic features to obtain unified semantic features. From the target semantic features, similar modal semantic features similar to the unified semantic features are selected; the similar modal semantic features and the unified semantic features are both temporal semantic features; cross-modal attention alignment is performed on the unified semantic features and the similar modal semantic features to obtain backup semantic features; other semantic features in the target semantic features are determined as heterogeneous modal semantic features corresponding to the backup semantic features; the backup semantic features and the heterogeneous modal semantic features are fused to generate candidate semantic features; the candidate semantic features are fused with the semantic features of the prompt words corresponding to the target prompt words to generate the target fused features.

12. A target vehicle, characterized in that, include: An electronic device and a vehicle body, the electronic device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the large model usage method based on multimodal data as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of using large models based on multimodal data as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of using large models based on multimodal data as described in any one of claims 1 to 10.

Images