Intelligent driving scene understanding and decision-making method and system based on multi-modal large language model

Abstract

The invention discloses an intelligent driving scene understanding and decision-making method and system based on a multi-modal large language model, and relates to the technical field of intelligent driving scene understanding and decision-making, and the method comprises the steps: collecting the visual, radar, laser radar, Internet of Vehicles, voice and vehicle state data of the surrounding environment of a vehicle, and forming a multi-modal original input set; performing feature extraction and semantic coding on various data in the multi-modal original input set to generate semantic feature vectors of corresponding modals; unified space mapping is carried out on the semantic feature vectors through a cross-modal alignment mechanism, multi-modal fusion processing is carried out based on an alignment result, and comprehensive semantic representation is generated; analyzing a social interaction relationship in the traffic scene based on the comprehensive semantic representation, identifying action modes and behavior tendencies of surrounding traffic participants, and generating social intention description information; and generating a scene query request according to the comprehensive semantic representation, and matching related traffic rules and driving experience in a pre-constructed driving common knowledge base.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving scene understanding and decision-making technology, and in particular to an intelligent driving scene understanding and decision-making method and system based on a multimodal large language model. Background Technology

[0002] Intelligent driving scenario understanding and decision-making technology refers to the technical system in which autonomous driving systems perceive environmental information through multiple sensors, perform semantic analysis of complex traffic scenarios, identify the behavioral intentions of traffic participants, and generate safe, reasonable, and compliant vehicle control strategies based on rules, experience, or learning models. Therefore, how to utilize advanced technologies to improve the intelligence level and safety of intelligent driving scenario understanding and decision-making has become one of the most pressing issues to be addressed.

[0003] In the field of intelligent driving scenario understanding and decision-making, traditional methods rely solely on trajectory prediction behavior, ignoring social signals such as pedestrian / non-motorized vehicle line of sight, posture changes, and group interactions. This makes it difficult to accurately judge psychological states such as distraction and hesitation. Furthermore, while large language models can generate natural language decisions, they suffer from problems such as illusions, logical errors, and unstable outputs, posing significant safety hazards if directly applied to vehicle control. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent driving scene understanding and decision-making method based on a multimodal large language model. This solves the problems of traditional methods that rely solely on trajectory prediction behavior, neglecting social signals such as pedestrian / non-motorized vehicle line of sight, posture changes, and group interactions, making it difficult to accurately judge psychological states such as distraction and hesitation. Furthermore, although large language models can generate natural language decisions, they suffer from problems such as illusions, logical errors, and unstable output, posing significant safety hazards when directly used for vehicle control.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an intelligent driving scene understanding and decision-making method based on a multimodal large language model, comprising:

[0008] Collect visual, radar, lidar, vehicle-to-everything (V2X) data, voice data, and vehicle status data of the vehicle's surrounding environment to form a multimodal raw input set;

[0009] Feature extraction and semantic encoding are performed on each type of data in the multimodal original input set to generate semantic feature vectors for the corresponding modalities;

[0010] The semantic feature vectors are mapped to a unified space through a cross-modal alignment mechanism, and multimodal fusion processing is performed based on the alignment results to generate a comprehensive semantic representation.

[0011] Based on comprehensive semantic representation analysis of social interaction relationships in traffic scenarios, the action patterns and behavioral tendencies of surrounding traffic participants are identified, and social intention description information is generated.

[0012] Based on the comprehensive semantic representation, a scenario query request is generated, and relevant traffic rules and driving experience are matched in the pre-built driving common sense knowledge base. The search results are then transformed into semantically enhanced information.

[0013] Social intention description information and semantic enhancement information are jointly input into a multimodal large language model to generate driving decision content. At the same time, the stability and consistency of the model's reasoning process are evaluated, and the decision confidence level is output.

[0014] The reliability of the current decision is judged based on the decision confidence level. When the reliability is insufficient, a conservative driving strategy is triggered. When the reliability meets the requirements, the decision content is converted into vehicle control commands and executed.

[0015] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the specific steps for extracting features and semantically encoding various types of data in the multimodal original input set to generate semantic feature vectors corresponding to the respective modalities are as follows:

[0016] After performing image distortion correction and illumination normalization on the visual data, it is input into the visual Transformer network. Through the self-attention mechanism, the network extracts the road boundaries, traffic signs, dynamic targets and their spatial distribution features in the image layer by layer, and outputs feature vectors that represent the visual semantics of the environment.

[0017] After performing ground segmentation and noise filtering on the lidar point cloud, three-dimensional target clustering is performed in conjunction with the radar target list to extract the geometric contours, motion speed and trajectory trends of the targets, which are then input into the PointNet network to generate feature vectors representing the spatial structure.

[0018] After performing noise reduction and voice endpoint detection on the speech data, the speech data is converted into text information through an automatic speech recognition model. Then, a pre-trained text encoder is used to extract the semantic intent in the text and generate a feature vector representing the driver's instructions.

[0019] The vehicle network data is parsed and spatiotemporally aligned to extract the positions of surrounding vehicles, traffic light phases and road event information. After applying a moving average filter to the vehicle state data, the vehicle's speed, acceleration and steering angle are extracted. The two data are then input into a fully connected network and mapped into a low-dimensional vector to generate a feature vector representing the vehicle's operating state.

[0020] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the steps of unifying spatial mapping of semantic feature vectors through a cross-modal alignment mechanism and performing multimodal fusion processing based on the alignment results to generate a comprehensive semantic representation are as follows:

[0021] Visual semantic feature vectors, spatial geometric feature vectors, speech semantic feature vectors, and structured state feature vectors are input into a cross-modal contrastive learning network. By optimizing the loss function, positive sample feature vectors describing the same traffic scenario are brought closer to each other in the vector space, while negative sample feature vectors describing different scenarios are moved further apart, thus achieving spatial alignment.

[0022] The aligned feature vectors of each modality are input into the multi-head cross-attention fusion module. The visual semantic feature vector is used as the query vector, and the spatial geometric feature vector, speech semantic feature vector, and structured state feature vector are used as the key and value vectors, respectively. The attention weights between each modality are calculated using the following expression:

[0023]

[0024] Where, α j Let m be the attention weight for the j-th non-visual modality. j Let be the semantic feature vector of the j-th non-visual modality, taken from spatial geometry, speech semantics, or structured state feature vectors; v is the visual semantic feature vector, used as the query vector; sim(.) represents the cosine similarity function; and M is the set of non-visual modalities.

[0025] The feature vectors of each modality are weighted and fused according to the attention weights to generate a joint feature vector containing visual, spatial, speech and state information.

[0026] Normalization is performed on the joint feature vector to generate a comprehensive semantic representation;

[0027] The posture key point coordinates and motion direction vectors of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. Nodes in the graph represent traffic participants, and the edge weights represent the attention intensity between individuals. The expression is as follows:

[0028]

[0029] Among them, a i Let d be the attention intensity of the i-th traffic participant. i Its line-of-sight vector, r i Let || be the relative direction vector between itself and the position of the vehicle, and || represent the magnitude of the vector.

[0030] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the specific steps of analyzing social interaction relationships in traffic scenes based on comprehensive semantic representation, identifying the action patterns and behavioral tendencies of surrounding traffic participants, and generating social intention description information are as follows:

[0031] The coordinates of the posture key points and the direction vector of movement of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. In the graph, nodes represent traffic participants and the weight of the edges represent the attention intensity between individuals.

[0032] By inputting the social attention graph into a graph neural network, the interaction patterns between individuals are analyzed through node feature updates and edge weight propagation mechanisms.

[0033] Based on the updated node features, the psychological state of each traffic participant is determined, and semantic descriptions including states such as distraction, hesitation, preparation to cross or move away from the lane are generated.

[0034] By combining semantic description with environmental context information in comprehensive semantic representation, social intention description information is generated.

[0035] The integrated semantic representation input query generation subnetwork generates a natural language-form scenario query statement, with the expression:

[0036] q = f query (s);

[0037] Where q is the generated query statement, s is the comprehensive semantic representation, and f query To query the nonlinear mapping function that generates the subnetwork.

[0038] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the steps of generating a scene query request based on a comprehensive semantic representation, matching relevant traffic rules and driving experience in a pre-constructed driving common sense knowledge base, and converting the retrieval results into semantically enhanced information are as follows:

[0039] The scenario query statement is input into the driving common sense knowledge base, which contains traffic regulations, local driving practices, special road section traffic rules and environmental impact experience knowledge. The knowledge items related to the current scenario are retrieved through semantic similarity matching.

[0040] Semantic analysis is performed on the retrieved knowledge entries to extract applicable conditions and suggested behaviors;

[0041] The applicable conditions and recommended behaviors are transformed into natural language descriptions to generate semantically enhanced information that includes content such as slowing down to a safe speed on construction sites in rainy weather or prohibiting honking in school zones.

[0042] During model inference, multiple random samplings are performed to obtain multiple candidate decision outputs. The distribution differences of each candidate decision across behavioral categories are statistically analyzed to evaluate the inference stability. The expression is as follows:

[0043]

[0044] Where H is the behavioral distribution entropy, k b Let B be the number of times behavior category b appears in n samples, where n is the total number of samples, and B is the set of all possible behavior categories.

[0045] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the steps of inputting social intention description information and semantic enhancement information into the multimodal large language model to generate driving decision content, simultaneously evaluating the stability and consistency of the model's reasoning process, and outputting the decision confidence level are as follows:

[0046] The prompting context is constructed by combining social intention description information, semantic enhancement information and environmental state information in the comprehensive semantic representation. The prompting context includes scene description, social risk warning and traffic rule constraints.

[0047] The context prompts are input into a multimodal large language model to generate driving decision content;

[0048] The distribution differences of each candidate decision across behavioral categories are statistically analyzed, and the behavioral distribution entropy is calculated to assess the stability of the reasoning process.

[0049] The consistency of the reasoning process is evaluated by combining the influence weights of each modality feature on the final decision.

[0050] The combined stability and consistency indices generate the decision confidence level, which is used to judge the reliability of the decision.

[0051] The decision confidence level is compared with a preset safety threshold to determine whether the current decision is reliable. The expression is:

[0052] Reliable={True,C≥T False,C <T;

[0053] Where Reliable represents the reliability assessment result, C represents the decision confidence level, and T represents the security threshold set by the system.

[0054] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the following steps are taken: The reliability of the current decision is judged based on the decision confidence level; when the reliability is insufficient, a conservative driving strategy is triggered; when the reliability meets the requirements, the decision content is converted into vehicle control commands and executed.

[0055] If the decision confidence level is lower than the safety threshold, it is determined that there is cognitive uncertainty in the current decision and a conservative driving strategy is initiated.

[0056] The conservative driving strategy includes maintaining the current lane, reducing the driving speed to a preset safe speed, prohibiting lane changes, and prompting the driver to take over vehicle control via the in-vehicle human-machine interface.

[0057] If the decision confidence level is not lower than the safety threshold, the driving decision content will be parsed into structured control instructions, which include target speed, target steering angle and acceleration limit parameters;

[0058] The structured control commands are converted into communication signals that can be recognized by the vehicle control bus, driving the vehicle to perform corresponding driving operations.

[0059] Receive abnormal scenario response cases uploaded by a group of vehicles, extract the verified and effective driving behavior patterns, and maintain the credibility of rules in the knowledge base. The expression is:

[0060]

[0061] Where R is the dynamic credibility of the knowledge entry, P is the initial confidence level, S is the number of successful historical verifications, N is the total number of calls, and F is the update frequency.

[0062] As a preferred embodiment of the intelligent driving scene understanding and decision-making method based on a multimodal large language model described in this invention, the driving common sense knowledge base is constructed and updated in the following manner:

[0063] Collect rule entries from national traffic regulations, local traffic management regulations, typical road design standards, and driver training materials, and structure them into entity-relation-entity triples to establish an initial knowledge base;

[0064] After manual review, valid patterns will be converted into knowledge entries and added to the knowledge base;

[0065] Receive temporary traffic control information, weather warnings, and road construction notices issued by traffic management departments through cloud services;

[0066] Automatically parse the announcement content, extract the affected road sections, time range and traffic requirements, and update the validity period and scope of application of relevant entries in the knowledge base;

[0067] In the process of multimodal fusion, it is necessary to evaluate the degree of support of each modality for the final decision, expressed as:

[0068]

[0069] Where, βj For the contribution weight of the j-th mode, m j Let be the semantic feature vector of the j-th modality, s be the comprehensive semantic representation, sim(.) be the cosine similarity, and W be the set of all modalities.

[0070] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in the first aspect of the present invention.

[0071] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in the first aspect of the present invention.

[0072] The beneficial effects of this invention are as follows: By constructing a collaborative architecture of multimodal feature extraction, cross-modal alignment and fusion, social interaction understanding and knowledge-enhanced reasoning, it significantly improves the understanding and decision-making capabilities of intelligent driving scenarios. It effectively solves the problems of difficulty in fusing multi-source heterogeneous information in complex traffic environments, inaccurate identification of traffic participants' intentions, poor interpretability of large model decisions and uncontrollable reliability. It enhances the cognitive robustness and behavioral safety of autonomous driving systems in scenarios with obstruction, severe weather and dense interaction. By introducing attention mechanisms, behavioral distribution entropy evaluation and dynamic knowledge credibility modeling, it not only improves the accuracy of semantic understanding and the rationality of decision-making, but also realizes quantitative monitoring and proactive defense of the output risks of large language models. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart of the intelligent driving scenario understanding and decision-making method based on a multimodal large language model in Example 1. Detailed Implementation

[0075] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0076] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0077] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0078] Example, refer to Figure 1 This embodiment of the invention provides a method for intelligent driving scene understanding and decision-making based on a multimodal large language model, comprising the following steps:

[0079] S1. Collect visual, radar, lidar, vehicle networking, voice, and vehicle status data of the vehicle's surrounding environment to form a multimodal raw input set;

[0080] Furthermore, visual data is continuously captured by forward-looking and panoramic cameras at a sampling frequency of no less than 30 frames per second, showing RGB images of the road scenes in front of and to the sides of the vehicle; radar data is obtained in real time by millimeter-wave radar, showing the distance, azimuth angle, and radial velocity information of target objects; lidar data is generated by multi-beam lidar scanning to produce high-precision 3D point clouds; vehicle-to-everything (V2X) data is received through the V2X communication module, showing the position, speed, and steering status of surrounding vehicles, as well as traffic light phases and road construction warnings issued by traffic infrastructure; voice data is collected through the in-vehicle microphone array, showing voice commands issued by the driver; and vehicle status data is read in real time through the CAN bus, showing vehicle speed, acceleration, steering angle, yaw rate, and braking status.

[0081] It should be noted that all types of sensor data in the multimodal raw input set are equipped with high-precision timestamps during acquisition. Sub-millisecond time synchronization is achieved through hardware-triggered synchronization or interpolation-based time alignment algorithms, ensuring that the data of different modalities are strictly aligned in the spatiotemporal dimension. This avoids semantic mismatch in environmental perception caused by differences in sampling delay, and provides a reliable data foundation for subsequent cross-modal fusion.

[0082] S2. Perform feature extraction and semantic encoding on each type of data in the multimodal original input set to generate semantic feature vectors for the corresponding modalities;

[0083] Furthermore, after performing image distortion correction and illumination normalization on the visual data, it is input into the visual Transformer network. Through the self-attention mechanism, the network extracts the road boundaries, traffic signs, dynamic targets and their spatial distribution features in the image layer by layer, and outputs feature vectors that represent the visual semantics of the environment.

[0084] After performing ground segmentation and noise filtering on the lidar point cloud, three-dimensional target clustering is performed in conjunction with the radar target list to extract the geometric contours, motion speed and trajectory trends of the targets, which are then input into the PointNet network to generate feature vectors representing the spatial structure.

[0085] After performing noise reduction and voice endpoint detection on the speech data, the speech data is converted into text information through an automatic speech recognition model. Then, a pre-trained text encoder is used to extract the semantic intent in the text and generate a feature vector representing the driver's instructions.

[0086] The vehicle network data is parsed and spatiotemporally aligned to extract the positions of surrounding vehicles, traffic light phases and road event information. After the vehicle state data is filtered by moving average, the vehicle speed, acceleration and steering angle are extracted. The two are input into a fully connected network and mapped into a low-dimensional vector to generate a feature vector representing the vehicle's operating state.

[0087] It should be noted that the feature extraction process for each modality is executed in parallel on a dedicated edge computing unit. Visual and LiDAR data are inferred using GPU acceleration, while voice and vehicle networking data are processed with low latency through an NPU. The semantic feature vectors output by each encoding branch have a unified embedding dimension, which facilitates cross-modal alignment and fusion in the shared vector space, thereby improving the overall real-time performance and computational efficiency of the system.

[0088] S3. The semantic feature vectors are mapped to a unified space through a cross-modal alignment mechanism, and multimodal fusion processing is performed based on the alignment results to generate a comprehensive semantic representation.

[0089] Furthermore, visual semantic feature vectors, spatial geometric feature vectors, speech semantic feature vectors, and structured state feature vectors are input into a cross-modal contrastive learning network. By optimizing the loss function, positive sample feature vectors describing the same traffic scene are brought closer to each other in the vector space, while negative sample feature vectors describing different scenes are moved further apart, thus achieving spatial alignment.

[0090] The aligned feature vectors of each modality are input into the multi-head cross-attention fusion module. The visual semantic feature vector is used as the query vector, and the spatial geometric feature vector, speech semantic feature vector, and structured state feature vector are used as the key and value vectors, respectively. The attention weights between each modality are calculated using the following expression:

[0091]

[0092] Where, α j Let m be the attention weight for the j-th non-visual modality. j Let be the semantic feature vector of the j-th non-visual modality, taken from spatial geometry, speech semantics, or structured state feature vectors; v is the visual semantic feature vector, used as the query vector; sim(.) represents the cosine similarity function; and M is the set of non-visual modalities.

[0093] The feature vectors of each modality are weighted and fused according to the attention weights to generate a joint feature vector containing visual, spatial, speech and state information.

[0094] Normalization is performed on the joint feature vector to generate a comprehensive semantic representation;

[0095] The posture key point coordinates and motion direction vectors of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. Nodes in the graph represent traffic participants, and the edge weights represent the attention intensity between individuals. The expression is as follows:

[0096]

[0097] Among them, a i Let d be the attention intensity of the i-th traffic participant. i Its line-of-sight vector, r i Let || be the direction vector relative to the position of the vehicle, and || represent the magnitude of the vector.

[0098] It should be noted that the cross-modal alignment mechanism achieves the comparability of semantics of each modality in a unified vector space through contrastive learning, while the multi-head cross-attention fusion module can dynamically allocate the supplementary weights of non-visual modalities to the visually dominant representation. Combined with the construction of the social attention map, it not only achieves deep fusion of multi-source information, but also provides interpretable interaction relationship priors for subsequent social intention recognition.

[0099] S4. Based on comprehensive semantic representation, analyze the social interaction relationships in traffic scenarios, identify the action patterns and behavioral tendencies of surrounding traffic participants, and generate social intention description information.

[0100] Furthermore, the coordinates of the posture key points and the direction vector of movement of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. In the graph, nodes represent traffic participants and the weights of the edges represent the attention intensity between individuals.

[0101] By inputting the social attention graph into a graph neural network, the interaction patterns between individuals are analyzed through node feature updates and edge weight propagation mechanisms.

[0102] Based on the updated node features, the psychological state of each traffic participant is determined, and semantic descriptions including states such as distraction, hesitation, preparation to cross or move away from the lane are generated.

[0103] By combining semantic description with environmental context information in comprehensive semantic representation, social intention description information is generated.

[0104] The integrated semantic representation input query generation subnetwork generates a natural language-form scenario query statement, with the expression:

[0105] q = f query (s);

[0106] Where q is the generated query statement, s is the comprehensive semantic representation, and f query To query the nonlinear mapping function that generates the subnetwork;

[0107] It should be noted that the generation of social intention description information relies on the topological structure analysis of the social attention graph by the graph neural network, which can capture the group behavior patterns and psychological state propagation effects among pedestrians, and generate high-order semantic descriptions in combination with environmental context, thereby improving the accuracy and foresight of predicting the behavior of unstructured traffic participants.

[0108] S5. Generate a scenario query request based on the comprehensive semantic representation, match relevant traffic rules and driving experience in the pre-built driving common sense knowledge base, and transform the retrieval results into semantically enhanced information;

[0109] Furthermore, the scenario query is input into the driving common sense knowledge base, which contains traffic regulations, local driving practices, special road section traffic rules and environmental impact experience knowledge. The knowledge items related to the current scenario are retrieved through semantic similarity matching.

[0110] Semantic analysis is performed on the retrieved knowledge entries to extract applicable conditions and suggested behaviors;

[0111] The applicable conditions and recommended behaviors are transformed into natural language descriptions to generate semantically enhanced information that includes content such as slowing down to a safe speed on construction sites in rainy weather or prohibiting honking in school zones.

[0112] During model inference, multiple random samplings are performed to obtain multiple candidate decision outputs. The distribution differences of each candidate decision across behavioral categories are statistically analyzed to evaluate the inference stability. The expression is as follows:

[0113]

[0114] Where H is the behavioral distribution entropy, k b Let B be the number of times behavior category b appears in n samples, where n is the total number of samples, and B is the set of all possible behavior categories.

[0115] The driving common sense knowledge base is built and updated in the following ways:

[0116] Collect rule entries from national traffic regulations, local traffic management regulations, typical road design standards, and driver training materials, and structure them into entity-relation-entity triples to establish an initial knowledge base;

[0117] After manual review, valid patterns will be converted into knowledge entries and added to the knowledge base;

[0118] Receive temporary traffic control information, weather warnings, and road construction notices issued by traffic management departments through cloud services;

[0119] Automatically parse the announcement content, extract the affected road sections, time range and traffic requirements, and update the validity period and scope of application of relevant entries in the knowledge base;

[0120] In the process of multimodal fusion, it is necessary to evaluate the degree of support of each modality for the final decision, expressed as:

[0121]

[0122] Where, β j For the contribution weight of the j-th mode, m j Let be the semantic feature vector of the j-th modality, s be the comprehensive semantic representation, sim(.) be the cosine similarity, and W be the set of all modalities;

[0123] It should be noted that the driving common sense knowledge base supports online dynamic updates and a credibility evolution mechanism. By introducing behavioral distribution entropy to evaluate the stability of large model reasoning and combining modal contribution weights to analyze the consistency of decision-making basis, it realizes the transformation from black-box decision-making to quantifiable and credible decision-making, providing dual protection for knowledge retrieval and risk warning in complex scenarios.

[0124] S6. Input the social intention description information and semantic enhancement information into the multimodal large language model to generate driving decision content, and at the same time evaluate the stability and consistency of the model's reasoning process and output the decision confidence level.

[0125] Furthermore, the social intention description information, semantic enhancement information, and environmental state information in the comprehensive semantic representation are jointly used to construct the prompt context, which includes scene description, social risk warning, and traffic rule constraints.

[0126] The context prompts are input into a multimodal large language model to generate driving decision content;

[0127] The distribution differences of each candidate decision across behavioral categories are statistically analyzed, and the behavioral distribution entropy is calculated to assess the stability of the reasoning process.

[0128] The consistency of the reasoning process is evaluated by combining the influence weights of each modality feature on the final decision.

[0129] The combined stability and consistency indices generate the decision confidence level, which is used to judge the reliability of the decision.

[0130] The decision confidence level is compared with a preset safety threshold to determine whether the current decision is reliable. The expression is:

[0131] Reliable={True,C≥T False,C <T;

[0132] Where Reliable is the reliability judgment result, C is the decision confidence level, and T is the security threshold set by the system;

[0133] It should be noted that the generation of decision confidence level integrates the dual indicators of model output stability and multimodal support consistency. The comparison result with the safety threshold directly determines whether the control command is executed, ensuring that when the cognitive uncertainty is high, it actively degrades to a conservative strategy, effectively preventing the large language model from generating hallucinatory or high-risk commands, and ensuring the functional safety of the autonomous driving system.

[0134] S7. Determine the reliability of the current decision based on the decision confidence level. If the reliability is insufficient, trigger a conservative driving strategy. If the reliability meets the requirements, convert the decision content into vehicle control commands and execute them.

[0135] Furthermore, if the decision confidence level is lower than the safety threshold, it is determined that there is cognitive uncertainty in the current decision and a conservative driving strategy is initiated.

[0136] Conservative driving strategies include maintaining the current lane, reducing the driving speed to a preset safe speed, prohibiting lane changes, and prompting the driver to prepare to take over vehicle control through the in-vehicle human-machine interface.

[0137] If the decision confidence level is not lower than the safety threshold, the driving decision content will be parsed into structured control instructions, which include target speed, target steering angle and acceleration limit parameters.

[0138] The structured control commands are converted into communication signals that can be recognized by the vehicle control bus, driving the vehicle to perform corresponding driving operations.

[0139] Receive abnormal scenario response cases uploaded by a group of vehicles, extract the verified and effective driving behavior patterns, and maintain the credibility of rules in the knowledge base. The expression is:

[0140]

[0141] Where R is the dynamic credibility of the knowledge item, P is the initial confidence level, S is the number of successful historical verifications, N is the total number of calls, and F is the update frequency;

[0142] It should be noted that the conservative driving strategy has a higher priority than the proactive decision-making generated by the large model, and the knowledge base credibility update mechanism continuously improves the system's ability to cope with long-tail scenarios by introducing a swarm intelligence feedback loop.

[0143] This embodiment also provides a computer device applicable to the intelligent driving scene understanding and decision-making method based on a multimodal large language model, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the intelligent driving scene understanding and decision-making method based on a multimodal large language model as proposed in the above embodiment.

[0144] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0145] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the intelligent driving scenario understanding and decision-making method based on a multimodal large language model as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0146] In summary, this invention significantly enhances the understanding and decision-making capabilities of intelligent driving scenarios by constructing a collaborative architecture that integrates multimodal feature extraction, cross-modal alignment and fusion, social interaction understanding, and knowledge-enhanced reasoning. It effectively addresses the challenges of multi-source heterogeneous information fusion in complex traffic environments, inaccurate identification of traffic participant intentions, poor interpretability of large model decisions, and uncontrollable reliability. This enhances the cognitive robustness and behavioral safety of autonomous driving systems in occluded, adverse weather, and densely interactive scenarios. By introducing attention mechanisms, behavioral distribution entropy assessment, and dynamic knowledge credibility modeling, it not only improves semantic understanding accuracy and decision rationality but also enables quantitative monitoring and proactive defense against risks in the output of large language models.

[0147] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent driving scene understanding and decision-making based on a multimodal large language model, characterized in that: include: Collect visual, radar, lidar, vehicle-to-everything (V2X) data, voice data, and vehicle status data of the vehicle's surrounding environment to form a multimodal raw input set; Feature extraction and semantic encoding are performed on each type of data in the multimodal original input set to generate semantic feature vectors for the corresponding modalities; The semantic feature vectors are mapped to a unified space through a cross-modal alignment mechanism, and multimodal fusion processing is performed based on the alignment results to generate a comprehensive semantic representation. Based on comprehensive semantic representation analysis of social interaction relationships in traffic scenarios, the action patterns and behavioral tendencies of surrounding traffic participants are identified, and social intention description information is generated. Based on the comprehensive semantic representation, a scenario query request is generated, and relevant traffic rules and driving experience are matched in the pre-built driving common sense knowledge base. The search results are then transformed into semantically enhanced information. Social intention description information and semantic enhancement information are jointly input into a multimodal large language model to generate driving decision content. At the same time, the stability and consistency of the model's reasoning process are evaluated, and the decision confidence level is output. The reliability of the current decision is judged based on the decision confidence level. When the reliability is insufficient, a conservative driving strategy is triggered. When the reliability meets the requirements, the decision content is converted into vehicle control commands and executed.

2. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 1, characterized in that: The specific steps for extracting features and semantically encoding various types of data in the multimodal original input set to generate semantic feature vectors for the corresponding modalities are as follows: After performing image distortion correction and illumination normalization on the visual data, it is input into the visual Transformer network. Through the self-attention mechanism, the network extracts the road boundaries, traffic signs, dynamic targets and their spatial distribution features in the image layer by layer, and outputs feature vectors that represent the visual semantics of the environment. After performing ground segmentation and noise filtering on the lidar point cloud, three-dimensional target clustering is performed in conjunction with the radar target list to extract the geometric contours, motion speed and trajectory trends of the targets, which are then input into the PointNet network to generate feature vectors representing the spatial structure. After performing noise reduction and voice endpoint detection on the speech data, the speech data is converted into text information through an automatic speech recognition model. Then, a pre-trained text encoder is used to extract the semantic intent in the text and generate a feature vector representing the driver's instructions. The vehicle network data is parsed and spatiotemporally aligned to extract the positions of surrounding vehicles, traffic light phases and road event information. After applying a moving average filter to the vehicle state data, the vehicle's speed, acceleration and steering angle are extracted. The two data are then input into a fully connected network and mapped into a low-dimensional vector to generate a feature vector representing the vehicle's operating state.

3. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 2, characterized in that: The steps for unifying spatial mapping of semantic feature vectors through a cross-modal alignment mechanism and performing multimodal fusion processing based on the alignment results to generate a comprehensive semantic representation are as follows: Visual semantic feature vectors, spatial geometric feature vectors, speech semantic feature vectors, and structured state feature vectors are input into a cross-modal contrastive learning network. By optimizing the loss function, positive sample feature vectors describing the same traffic scenario are brought closer to each other in the vector space, while negative sample feature vectors describing different scenarios are moved further apart, thus achieving spatial alignment. The aligned feature vectors of each modality are input into the multi-head cross-attention fusion module. The visual semantic feature vector is used as the query vector, and the spatial geometric feature vector, speech semantic feature vector, and structured state feature vector are used as the key and value vectors, respectively. The attention weights between each modality are calculated using the following expression: Where, α j Let m be the attention weight for the j-th non-visual modality. j Let be the semantic feature vector of the j-th non-visual modality, taken from spatial geometry, speech semantics, or structured state feature vectors; v is the visual semantic feature vector, used as the query vector; sim(.) represents the cosine similarity function; and M is the set of non-visual modalities. The feature vectors of each modality are weighted and fused according to the attention weights to generate a joint feature vector containing visual, spatial, speech and state information. Normalization is performed on the joint feature vector to generate a comprehensive semantic representation; The posture key point coordinates and motion direction vectors of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. Nodes in the graph represent traffic participants, and the edge weights represent the attention intensity between individuals. The expression is as follows: Among them, a i Let d be the attention intensity of the i-th traffic participant. i Its line-of-sight vector, r i Let || be the relative direction vector between itself and the position of the vehicle, and || represent the magnitude of the vector.

4. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 3, characterized in that: The steps for analyzing social interaction relationships in traffic scenarios based on comprehensive semantic representation, identifying the action patterns and behavioral tendencies of surrounding traffic participants, and generating social intention description information are as follows: The coordinates of the posture key points and the direction vector of movement of pedestrians or non-motorized vehicles are extracted from the comprehensive semantic representation to construct a social attention graph. In the graph, nodes represent traffic participants and the weight of the edges represent the attention intensity between individuals. By inputting the social attention graph into a graph neural network, the interaction patterns between individuals are analyzed through node feature updates and edge weight propagation mechanisms. Based on the updated node features, the psychological state of each traffic participant is determined, and semantic descriptions including states such as distraction, hesitation, preparation to cross or move away from the lane are generated. By combining semantic description with environmental context information in comprehensive semantic representation, social intention description information is generated. The integrated semantic representation input query generation subnetwork generates a natural language-form scenario query statement, with the expression: q=f query (s); Where q is the generated query statement, s is the comprehensive semantic representation, and f query To query the nonlinear mapping function that generates the subnetwork.

5. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 4, characterized in that: The steps for generating a scenario query request based on comprehensive semantic representation, matching relevant traffic rules and driving experience in a pre-built driving common sense knowledge base, and converting the retrieval results into semantically enhanced information are as follows: The scenario query is input into the driving common sense knowledge base, which contains traffic regulations, local driving practices, special road section traffic rules and environmental impact experience knowledge. The knowledge items related to the current scenario are retrieved through semantic similarity matching. Semantic analysis is performed on the retrieved knowledge entries to extract applicable conditions and suggested behaviors; The applicable conditions and recommended behaviors are transformed into natural language descriptions to generate semantically enhanced information that includes content such as slowing down to a safe speed on construction sites in rainy weather or prohibiting honking in school zones. During model inference, multiple random samplings are performed to obtain multiple candidate decision outputs. The distribution differences of each candidate decision across behavioral categories are statistically analyzed to evaluate the inference stability. The expression is as follows: Where H is the behavioral distribution entropy, k b Let B be the number of times behavior category b appears in n samples, where n is the total number of samples, and B is the set of all possible behavior categories.

6. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 5, characterized in that: The process involves inputting social intention description information and semantic enhancement information into a multimodal large language model to generate driving decision content, while simultaneously evaluating the stability and consistency of the model's reasoning process and outputting the decision confidence level. The specific steps are as follows: The prompting context is constructed by combining social intention description information, semantic enhancement information and environmental state information in the comprehensive semantic representation. The prompting context includes scene description, social risk warning and traffic rule constraints. The context prompts are input into a multimodal large language model to generate driving decision content; The distribution differences of each candidate decision across behavioral categories are statistically analyzed, and the behavioral distribution entropy is calculated to assess the stability of the reasoning process. The consistency of the reasoning process is evaluated by combining the influence weights of each modality feature on the final decision. The combined stability and consistency indices generate the decision confidence level, which is used to judge the reliability of the decision. The decision confidence level is compared with a preset safety threshold to determine whether the current decision is reliable. The expression is: Reliable={True,C≥T False,C <T; Where Reliable represents the reliability assessment result, C represents the decision confidence level, and T represents the security threshold set by the system.

7. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 6, characterized in that: The reliability of the current decision is determined based on the decision confidence level. When the reliability is insufficient, a conservative driving strategy is triggered. When the reliability meets the requirements, the decision content is converted into vehicle control commands and executed. The specific steps are as follows: If the decision confidence level is lower than the safety threshold, it is determined that there is cognitive uncertainty in the current decision and a conservative driving strategy is initiated. The conservative driving strategy includes maintaining the current lane, reducing the driving speed to a preset safe speed, prohibiting lane changes, and prompting the driver to take over vehicle control via the in-vehicle human-machine interface. If the decision confidence level is not lower than the safety threshold, the driving decision content will be parsed into structured control instructions, which include target speed, target steering angle and acceleration limit parameters; The structured control commands are converted into communication signals that can be recognized by the vehicle control bus, driving the vehicle to perform corresponding driving operations. Receive abnormal scenario response cases uploaded by a group of vehicles, extract the verified and effective driving behavior patterns, and maintain the credibility of rules in the knowledge base. The expression is: Where R is the dynamic credibility of the knowledge entry, P is the initial confidence level, S is the number of successful historical verifications, N is the total number of calls, and F is the update frequency.

8. The intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in claim 7, characterized in that: The driving common sense knowledge base is constructed and updated in the following ways: Collect rule entries from national traffic regulations, local traffic management regulations, typical road design standards, and driver training materials, and structure them into entity-relation-entity triples to establish an initial knowledge base; After manual review, valid patterns will be converted into knowledge entries and added to the knowledge base; Receive temporary traffic control information, weather warnings, and road construction notices issued by traffic management departments through cloud services; Automatically parse the announcement content, extract the affected road sections, time range and traffic requirements, and update the validity period and scope of application of relevant entries in the knowledge base; In the process of multimodal fusion, it is necessary to evaluate the degree of support of each modality for the final decision, expressed as: Where, β j For the contribution weight of the j-th mode, m j Let be the semantic feature vector of the j-th modality, s be the comprehensive semantic representation, sim(.) be the cosine similarity, and W be the set of all modalities.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent driving scene understanding and decision-making method based on a multimodal large language model as described in any one of claims 1 to 8.

Images