Method for semantic aware coordination of autonomous driving vehicles

Abstract

The invention discloses a semantic perception coordination method for automatic driving vehicles, and the method comprises the steps: collecting the state information of vehicles in real time, and dividing the vehicles into different temporary groups according to the spatial position relation between the vehicles; data synchronization and leader election are carried out in each group by adopting a Raft protocol, followers in each temporary group periodically upload own state information to the leader, and the leader generates a consistency state matrix of the group according to the collected state information of the followers in the group; the leader in each group broadcasts the consistency state matrix to all followers and inputs the consistency state matrix into a large language model, the consistency state matrix is converted into natural language description through the large language model, and action parameters of each group member are generated; and each vehicle inputs the action parameters into the rule-based driving model, and the rule-based driving model generates a control instruction according to the state information of the vehicle, so that the motion consistency of the vehicles in the group is realized. According to the invention, semantic perception and consistency coordination between vehicles are realized, and the safety and reliability of an intelligent traffic system are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for semantic perception coordination of autonomous vehicles. Background Technology

[0002] With the rapid development of Intelligent Transportation Systems (ITS), autonomous driving technology has become an important component of future smart mobility. Currently, autonomous vehicles rely heavily on onboard sensors (such as LiDAR, cameras, and millimeter-wave radar) for environmental perception and path planning. However, due to the limited field of view of individual vehicle sensors, blind spots are easily created, leading to limitations in the vehicle's understanding of its surroundings and thus posing safety hazards.

[0003] To compensate for the limitations of single-vehicle perception, centralized communication architectures such as Vehicle-to-Everything (V2X) have been gradually introduced, aiming to achieve global environmental perception and path optimization through information sharing between vehicles and between vehicles and infrastructure. However, existing centralized V2X networks generally suffer from the risk of Single Point of Failure (SPOF). Once the central server or communication node fails, it may lead to the interruption of information transmission between vehicles, seriously affecting traffic safety and driving efficiency. In addition, while some current distributed consensus methods, such as blockchain, Paxos, and PBFT, have achieved certain applications in data consistency and security, they mostly focus on the data transmission integrity and encryption security at the network protocol level, lacking the perception and understanding of vehicle behavior semantics and contextual information, and making it difficult to effectively distinguish traffic priorities in different scenarios (such as obstacle avoidance information and lane change requests).

[0004] In recent years, Large Language Models (LLMs) have been increasingly applied to fields such as robot control, natural language interaction, and intelligent planning in open scenarios due to their powerful semantic understanding and reasoning capabilities. However, existing LLM-driven autonomous driving methods are mostly based on single-vehicle architectures, lacking effective inter-vehicle collaboration mechanisms, and thus failing to fully leverage the global safety and traffic efficiency improvements brought about by multi-vehicle cooperation.

[0005] Therefore, there is an urgent need to propose a method and system for autonomous vehicle collaboration that combines the semantic understanding capabilities of large language models with distributed consensus protocols, so as to simultaneously achieve semantic perception, data consistency and stable and efficient group decision-making among vehicles, thereby improving the safety and reliability of intelligent transportation systems. Summary of the Invention

[0006] The purpose of this application is to provide a method for semantic perception coordination of autonomous vehicles, so as to achieve semantic perception and consistency coordination among autonomous vehicles.

[0007] In a first aspect, this application provides a method for semantic perception coordination in autonomous vehicles, comprising:

[0008] Real-time collection of vehicle status information; vehicles are divided into different temporary groups based on their spatial relationships.

[0009] The Raft protocol is used for data synchronization and leader election within each group. Followers in each temporary group periodically upload their own state information to the leader. The leader generates a consistent state matrix for the group based on the collected state information of the followers in the group.

[0010] The leader within each group broadcasts the consensus state matrix to all followers and inputs it into a large language model, which then converts it into a natural language description and generates action parameters for each group member;

[0011] Each vehicle inputs the motion parameters into a rule-based driving model, which generates control commands based on the vehicle's own state information to achieve motion consistency among vehicles in the group.

[0012] In a preferred embodiment, dividing vehicles into different temporary groups based on their spatial relationships further includes:

[0013] Vehicles whose spatial distance from each other is less than or equal to a first preset threshold are grouped into one group.

[0014] In a preferred embodiment, the method further includes: when a vehicle is in a separate group, if the spatial distance between the vehicle and all vehicles in another group at the current moment is less than a preset merging threshold, then the vehicle is merged into that group.

[0015] In a preferred embodiment, the method further includes: if the spatial distance between a vehicle in a group and any other vehicle in the group at the current moment is greater than a preset splitting threshold, then splitting the vehicle into a separate group.

[0016] In a preferred embodiment, the preset merging threshold is less than the first preset threshold, and the first preset threshold is less than the preset splitting threshold.

[0017] In a preferred embodiment, the first preset threshold is 30 meters, the preset merging threshold is 20 meters, and the preset splitting threshold is 40 meters.

[0018] In a preferred embodiment, the motion consistency means that the velocity directions are substantially the same and the difference between the velocities is less than or equal to a second preset threshold.

[0019] In a preferred embodiment, the motion parameters include acceleration range and steering angle range.

[0020] In a preferred embodiment, the control commands include actual acceleration and steering angle values.

[0021] In a preferred embodiment, the vehicle's status information includes position coordinates, speed, acceleration, steering angle, road condition information, and the status of surrounding vehicles.

[0022] In a preferred embodiment, the traffic information includes lane information, traffic lights, and intersection distance.

[0023] In a preferred embodiment, the surrounding vehicle status includes the relative position, speed, and acceleration of neighboring vehicles.

[0024] In a preferred embodiment, it also includes:

[0025] Set all vehicles in the group as followers, and use a heartbeat mechanism to detect the current leader's status. If the current leader's heartbeat is not detected, a new leader election will be held.

[0026] If the number of times a leader fails to be elected exceeds a preset number, the group will be disbanded, and the vehicles in that group will be grouped and added to other groups.

[0027] In a preferred embodiment, the preset number of times is 3.

[0028] In a preferred embodiment, the method further includes: if the response time of the large language model exceeds a predetermined time, the leader is switched to a proportional-integral-derivative controller.

[0029] In a preferred embodiment, this also includes: leaders within each group setting different priorities based on different broadcast messages.

[0030] In a second aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the aforementioned method.

[0031] It should be understood that, within the scope of this invention, the above-described technical features of this invention and the technical features specifically described below (such as in the embodiments) can be combined with each other to form new or preferred technical solutions. Due to space limitations, they will not be described in detail here. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. It should be understood that the accompanying drawings described below are merely some implementation examples of the present invention, and those skilled in the art can obtain other implementation examples based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating a method for semantic perception coordination of autonomous vehicles according to one embodiment of this application.

[0034] Figure 2 A system block diagram of LLM-Craft in one embodiment of this application is shown.

[0035] Figure 3 The process of dynamic grouping in one embodiment of this application is illustrated.

[0036] Figure 4 Different test scenarios are shown in one embodiment of this application.

[0037] Figure 5 This paper illustrates the relationship between LLM smart scoring and performance metrics in one embodiment of this application.

[0038] Figure 6 This paper illustrates the relationship between LLM intelligent scoring and algorithm performance in one embodiment of this application. Detailed Implementation

[0039] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0040] Through extensive and in-depth research, the inventors have proposed a method for semantic perception coordination in autonomous vehicles. The method dynamically groups vehicles based on their proximity and achieves information sharing within each group using the Raft algorithm. Utilizing the vehicle's state information, a Large Language Model (LLM) is used to generate motion parameters for each group member, including acceleration and steering angle ranges. Each group member then generates actual parameter values ​​based on its own state information and motion parameters through an Intelligent Driving Model (IDM), thereby achieving global planning and motion consistency for the vehicle.

[0041] More specifically, this invention proposes a three-stage decentralized framework, LLM-CRaft, which integrates LLM with distributed consensus and introduces the following functions: (1) dynamic vehicle grouping for context-aware coordination; (2) consensus-driven data synchronization using the Raft protocol for reliable state sharing; and (3) hybrid control combining the action range generated by LLM with IDM-based refinement. The LLM-CRaft framework is the first distributed consensus framework that uses the Raft algorithm to embed semantic understanding into the coordination of autonomous vehicles. LLM is positioned as a semantic interpreter between the consensus layer and the control layer. Raw sensor data collected by the vehicle is converted into natural language descriptions to achieve semantic abstraction. Furthermore, LLM assigns urgency (priority) to different types of messages during the consensus process to prioritize them according to context. LLM assigns urgency to messages; for example, LLM can distinguish between "collision risk" and "lane change request." For identified emergency events (such as potential collisions), LLM assigns them higher priority.

[0042] This application has at least the following advantages:

[0043] Experiments conducted in four scenarios (ordinary roads, congested roads, highways, and intersections) demonstrate that LLM-CRaft significantly outperforms existing methods, improving completion time by 10% to 30%. LLM-CRaft also achieves perfect completion rates in most cases. Compared to non-consensus baselines, LLM-CRaft achieves a 20% reduction in collisions and a 2 m / s increase in average speed under congested conditions. Extensive testing across multiple LLMs has also shown that LLM-Craft exhibits superior scalability with increasing LLM intelligence, with even more significant performance improvements when using more advanced models. This work pioneers a new paradigm for trustworthy autonomous transportation through semantically aware distributed coordination.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] One embodiment of this application relates to a method for semantic perception coordination in autonomous vehicles, the process of which is as follows: Figure 1 As shown, the method includes the following steps:

[0046] Step 101: Collect vehicle status information in real time and divide the vehicles into different temporary clusters or groups based on their spatial relationships.

[0047] Status information can be obtained through various sensors integrated into the vehicle (such as LiDAR, cameras, millimeter-wave radar, etc.). Status information can also be referred to as sensor data, including vehicle information, surrounding vehicle status information, road information, etc. It should be noted that vehicles that are close to each other are usually grouped together, and the grouping is dynamic; a vehicle's group affiliation will change according to changes in the vehicle's status.

[0048] Step 102: Within each group, the Raft protocol is used for data synchronization and leader election. Followers in each temporary group periodically upload their own state information to the leader. The leader generates a consistent state matrix for the group based on the collected state information from the followers. It should be noted that in the Raft protocol, each group has only one leader, responsible for initiating and deciding all operations to coordinate control instructions. Followers receive instructions from the leader and perform the corresponding operations.

[0049] Step 103: The leader within each group broadcasts the consensus state matrix to all followers and inputs it into the LLM. The LLM then converts this matrix into a natural language description and generates motion parameters for each group member. Motion parameters may include acceleration range and / or steering angle range.

[0050] Step 104: Each vehicle inputs its motion parameters into a rule-based driving model. The rule-based intelligent driving model (e.g., IDM) calculates the actual acceleration and steering angle values ​​based on the vehicle's own state information, thereby generating the final control command to achieve motion consistency among vehicles in the group.

[0051] The leader generates semantic action parameters based on a consistent state matrix and broadcasts the results to all followers via the Raft protocol. Followers then execute corresponding operations based on the broadcast parameters. Based on the Raft protocol, each vehicle possesses consistent global information, thereby avoiding individual vehicle decision bias and increasing decision stability. This application achieves semantic-level consistency for autonomous vehicles through the Raft protocol and a large language model.

[0052] To better understand the technical solution of this application, specific examples are provided below. The details listed in these examples are for ease of understanding and are not intended to limit the scope of protection of this application.

[0053] like Figure 2As shown, the LLM-CRaft system framework adopts a three-level cascaded architecture to balance real-time response capabilities and coordinated intelligence. The system flow first involves dynamic vehicle grouping, organizing vehicles into context-aware clusters; then, consensus-driven data synchronization is performed to ensure information consistency within each group. Finally, the hybrid control phase leverages the semantic understanding of LLM while maintaining rule-based control stability.

[0054] The LLM-CRaft system of this invention embeds LLM into a Raft consensus state machine through dynamic vehicle grouping, consensus-driven data synchronization, and hybrid control. Vehicles form self-organizing clusters using distance thresholds and motion consistency. For example, vehicles whose spatial distance is less than or equal to a preset distance threshold, whose speed directions are basically the same (angles are basically the same or less than or equal to a predetermined angle difference threshold), and whose speed differences are less than or equal to a preset speed difference threshold are grouped into a group. The Raft algorithm promotes information sharing within each group, thereby enhancing decision stability. Then, the LLM outputs the acceleration range and steering angle range, which are converted into precise values ​​by the IDM algorithm to ensure stability. The steering angle refers to the angle of steering wheel rotation. When the steering wheel is turned, the angle of deflection of the steering wheel relative to its initial position (e.g., when the vehicle is traveling in a straight line) is the steering angle. The speed of steering wheel rotation (i.e., the angle the steering wheel rotates per unit time) is expressed as angular velocity.

[0055] Dynamic vehicle grouping

[0056] Static vehicle grouping cannot adapt to rapid traffic changes, while random grouping lacks contextual relevance. Dynamic methods address these limitations by continuously regrouping vehicles based on real-time spatial relationships, such as... Figure 3 As shown, the Area of ​​Interest (AOI) for groups and vehicles is predefined. A single vehicle will form its own group and merge or converge with other vehicles when it encounters them. When groups converge, smaller groups merge into larger groups, forming an extended group. Conversely, if vehicles are isolated, they may detach and form separate groups.

[0057] Let the set of vehicles be V = {v1, v2, ..., v...} n}, v1, v2, v n These represent the positions p of the 1st, 2nd, and nth vehicles in the group, respectively, at time t. i (t)∈R 2 p i (t) represents the position of vehicle i at time t, which is in a two-dimensional space (R represents a real number, Ri). 2 Represented in two-dimensional real space. In traffic scenarios, this typically refers to the vehicle's coordinates on a plane, such as (x, y) coordinates, where x and y are real numbers. Group It is dynamically generated using the following formula:

[0058]

[0059] A vehicle v i A vehicle belongs to (or is dynamically assigned to) a vehicle group Gk if and only if there is another vehicle v in that group Gk. j This makes the vehicle v u and v j The distance between them is less than or equal to D thresh D thresh This represents the preset distance threshold, which is set to 30m here. The 30-meter threshold balances communication reliability (shorter distances reduce packet loss) and coordination relevance (longer distances capture more situational vehicles).

[0060] Groups change dynamically through merging or splitting, following these two rules:

[0061] • Merge: If minv i ∈G a ,v j ∈G b :||p i -p j ||2<D merging =20m, vehicles v exist in groups Ga and Gb respectively. i v j The distance between them is less than D merging Then, the Ga group and the Gb group are merged to ensure that the groups can coordinate as early as possible when they come into contact with each other. Where D... merging It equals 20 meters.

[0062] • Split (or split): If minv j ∈G k :||p i -p j ||2>D division =40m, vehicle v i The distance to the nearest car in group Gk is greater than D. division In other words, the distance to any vehicle in group Gk is greater than D. division v i Leave group Gk to prevent communication delays caused by an excessively large group. Where D... division It equals 40 meters.

[0063] The merging threshold is less than the splitting threshold. This lag design can prevent group oscillations at the boundary distance.

[0064] It should be noted that the merging threshold is less than the preset distance threshold, which in turn is less than the splitting threshold. For example, the merging threshold can be set within a range of 15 to 25 meters, the distance threshold within a range of 25 to 35 meters, and the splitting threshold within a range of 35 to 45 meters. The difference between the two thresholds should be greater than 10 meters.

[0065] Consensus-driven data synchronization

[0066] The original Raft protocol is used to maintain proven security guarantees. The consensus process consists of three phases:

[0067] 1) Leader Election: The standard heartbeat mechanism is used to detect leader failure. If no heartbeat is received for a period of time, the followers will initiate a new election.

[0068] 2) Data Aggregation: Followers periodically send sensor data (velocity, acceleration, etc.) to the leader. The leader maintains a shared state matrix M∈R. n×d It contains n group members and d features.

[0069] 3) Decision replication: The leader broadcasts the planning decision D to all followers to ensure eventual consistency. This synchronized decision constitutes subsequent vehicle control.

[0070] By separating consensus from decision-making logic, network-level reliability is ensured while allowing for pluggable planning algorithms. An example algorithm for vehicle coordination consensus is as follows:

[0071] Algorithm 1Consensus for Vehicle Coordination

[0072] Require:data={V,a,ω,E}

[0073] Ensure Synchronization

[0074] 1: v←follower

[0075] 2:while leader do

[0076] 3: V--elect→vleader

[0077] 4:end while

[0078] 5:while leader_active do

[0079] 6: --vheartbeat--→vleader

[0080] 7:if heartbeat then

[0081] 8: re-elect(vleader)

[0082] 9:endif

[0083] 10:Vfollower--location--→vleader

[0084] 11:vleader--process--→LLM(data)

[0085] 12:LLM---decisions--→Vfollower

[0086] 13:end while

[0087] Hybrid Control Line

[0088] Pure LLM control suffers from numerical instability, while rule-based methods lack adaptability. The hybrid method of this invention combines the advantages of both:

[0089] Phase 1: Semantic contextualization. The leader vehicle translates the consensus state M into a natural language prompt.

[0090] Second stage: Constraint generation. The LLM outputs an interpretable range, rather than precise values.

[0091] Phase Three: IDM Refinement. The IDM translates these scopes into executable controls:

[0092]

[0093] in, The acceleration range is defined by: v = v0 / v0 ...

[0094] Example prompts

[0095] (a) LLM Role: You are a driver, driving a vehicle on the road. (b) Available Information: Current information about your vehicle: [Vehicle information: speed, acceleration, position coordinates, angular velocity, etc.] Information about surrounding vehicles: [Surrounding vehicle information: speed and coordinates of surrounding vehicles, number of surrounding vehicles, etc.] Information about the road you are driving on: [Environmental information: distance to intersections, road complexity, road length, traffic light timing, etc.] (c) Task Description: As a driver, you need to assess the current driving situation based on your vehicle's own information, surrounding vehicle information, and environmental information. (d) Output Format: Please consider acceleration as pressing the accelerator pedal and angular velocity as turning the steering wheel. Provide the acceleration and angular velocity that you think are suitable for the current scenario: [Acceleration range] & [Steering angle range]. (e) Expected Output: [Acceleration range] [Steering angle range] Fault Tolerance Mechanism

[0096] Practical deployment of LLM-CRaft requires robustness to cope with hardware failures and algorithmic uncertainties. Therefore, this invention employs the following two fault-tolerance mechanisms:

[0097] • Backoff planner: If the LLM response delay exceeds the threshold, the proportional-integral-derivative (PID) controller takes over.

[0098] Consensus timeout: If the leader election fails for three consecutive cycles, the group is automatically disbanded, allowing vehicles to rejoin the functional group.

[0099] To rigorously verify the effectiveness and advantages of the LLM-CRaft mechanism, this invention constructs the experiment into four parts: main results and other parts progressing sequentially from inter-group coordination to scalability.

[0100] 1. Main Results Analysis: A comprehensive comparison of LLM-CRaft with existing LLM control methods was conducted in five state-of-the-art LLM systems and four driving scenarios.

[0101] 2. Inter-group coordination: Test the coordination between different vehicle groups to evaluate the system coordination and efficiency under LLM-CRaft compared to a non-consensus baseline.

[0102] 3. Intra-group coordination: Evaluate collaboration within a single group to test the LLM-CRaft mechanism at a granular level.

[0103] 4. Ablation study: Test the system by omitting key components such as combination and merging, leader selection and synchronization to demonstrate the importance of each step.

[0104] 5. Scalability Analysis: Demonstrates how the LLM-Craft approach scales as LLM functionality advances.

[0105] This invention constructs the following various driving environments to cover various scenarios encountered by vehicles under normal driving conditions:

[0106] • Normal roads: Sparse traffic, with few vehicles on the road;

[0107] • Crowded roads: Busy roads with high traffic volume;

[0108] • Highway: A road where vehicles travel at high speeds;

[0109] • Intersection: A complex intersection where vehicle movement is complex.

[0110] Figure 4 The four scenarios shown comprehensively demonstrate the various challenges that vehicles face in real-world driving situations.

[0111] This invention tested four high-performance driving methods based on LLM, including a zero-shot LLM method for comparison, and Drive, DiLu, and LLMLight. The performance of different LLM-based high-performance driving methods was tested on different model architectures (e.g., Gemini-2 GPT-3.5-Turbo, GPT-4-Turbo, LlaMA-3.2-90B, Deepseek-v3-671B). Each method was executed 30 times. The test results are shown in Table 1. Table 1 shows a comparison of the performance (in terms of completion time) of different methods under different models and driving scenarios, in seconds, and indicates the percentage of completion (i.e., completion rate). The LLM-CRaft method outperforms most other methods, especially in complex scenarios, with a higher completion rate and shorter completion time. Each character in “NHCI” represents ordinary road, highway, congested road, and intersection, respectively. The LLM-CRaft algorithm demonstrates strong collaborative capabilities, especially in complex road conditions, and outperforms some state-of-the-art (SOTA) methods in collective driving, while achieving 10-30% faster start-up speeds.

[0112]

[0113] Table 1. Performance comparison of different methods under different models and driving scenarios.

[0114] Table 1 comprehensively compares the performance of LLM-Craft with existing methods under five state-of-the-art LLM models and four different driving scenarios. The results show that LLM-Craft consistently outperforms competing methods under various driving conditions, highlighting the robustness and adaptability of the LLM-CRaft mechanism.

[0115] Among all evaluated LLM models, the LLM-CRaft method achieved superior performance metrics, with a 10-30% improvement in completion time compared to the baseline.

[0116] Specifically, compared to the suboptimal method (LLMLight), the LLM-CRaft method reduces completion time by an average of 7-15%, and by up to 25% compared to zero-shot LLM implementations. Notably, these performance improvements are maintained across a variety of model architectures and capabilities, from the smaller GPT-3.5-Turbo to more powerful models (e.g., DeepSeek-V3-671B), demonstrating that the method of this invention effectively utilizes the underlying capabilities of each model while mitigating individual limitations through collective intelligence.

[0117] The performance advantage of the LLM-CRaft method is particularly evident in complex driving scenarios. In congested environments, the method of this invention achieves a perfect completion rate (1.00) on all LLMs, with an average completion time 9-20% faster than other competing methods. Similarly, at intersections—a challenging scenario that typically requires complex coordination—the LLM-CRaft method achieves a completion rate between 0.90 and 0.99, outperforming all benchmark methods.

[0118] Highway scenarios require continuous high-speed coordination and rapid decision-making, revealing the most significant differences between various methods. Here, the LLM-CRaft method achieves a completion rate between 0.88 and 0.95, which is 1-2% higher than the suboptimal method and up to 14% higher than zero-sample LLM. This performance gap highlights the effectiveness of the distributed LLM-CRaft mechanism in maintaining coordination stability under harsh conditions.

[0119] A key conclusion drawn from these results is that the performance patterns of all LLM architectures remain consistent. While the functionality of a single LLM certainly impacts absolute performance metrics, the relative improvements achieved by the LLM-CRaft method remain remarkably stable. This demonstrates that the method of this invention successfully abstracts the coordination layer from model-specific constraints, thereby effectively creating an adaptive framework capable of leveraging any underlying LLM.

[0120] The results from DeepSeek-V3-671B are particularly noteworthy because the method of this invention achieves a near-perfect completion rate (0.99%) even in complex intersection scenarios, demonstrating that the LLM-CRaft mechanism can effectively utilize the capabilities of cutting-edge models. Similarly, compared to LLMLight, the method of this invention reduces the intersection completion time by 0.3 seconds on GPT-4-Turbo while maintaining a higher completion rate, highlighting the efficiency improvements brought about by the distributed coordination strategy.

[0121] These results validate the hypothesis of this invention: regardless of the underlying LLM, a well-designed LLM-CRaft mechanism can significantly enhance multi-agent coordination capabilities in autonomous driving scenarios. Consistent performance advantages across all evaluation dimensions demonstrate that the method of this invention effectively addresses the fundamental challenges of distributed decision-making in dynamic, safety-critical environments.

[0122] This invention conducted inter-group experiments to investigate the performance of different groups under the same settings and initial conditions. To evaluate the advantages of LLM-CRaft, this invention created a scenario with a fixed group configuration. Group 1 had 10-13 vehicles, Group 2 had approximately 10 vehicles, and this was adjusted according to the environment. In this scenario, the vehicles traveled closely together, and the groups were spaced far apart to prevent any groups from merging or splitting.

[0123] After initiating dynamic grouping, speed, acceleration, and collision rate were tested under both LLM-CRaft and non-LLM-Craft conditions, with 30 tests per configuration. Under the same scene and grouping conditions, performance with and without LLM-CRaft was compared, and the impact on Group 1 and Group 2 was analyzed separately. The results show that each group exhibits a significant advantage under the method of this invention. Table 2 shows the inter-group average speed, average collision rate, and average acceleration with and without LLM-CRaft in four environments. The results indicate that LLM-CRaft reduces the collision rate and increases the speed, as shown in Table 2, suggesting superior performance and stability.

[0124]

[0125]

[0126] Table 2 Comparison of results with and without LLM-CRaft in different environments.

[0127] In the intra-group experiment, 10 vehicles were randomly selected from a group of 30 vehicles and tested in four different scenarios to observe the intra-group performance of LLM-CRaft. Each vehicle used the same conditions but different positions, and their respective trajectories and interactions were tracked throughout the simulation. For vehicles using LLM-CRaft, LLM-based decision-making integrates information from all members of the group, while vehicles not using LLM-CRaft make decisions based solely on their local observations. The use and non-use of LLM-CRaft were compared for the 10 randomly selected vehicles under different driving environments. Each configuration was tested for 30 rounds. As shown in Table 3, LLM-CRaft enabled these vehicles to achieve a speed advantage of over 2 m / s, reduce the collision rate by approximately 20%, and increase acceleration by 1 to 3 m / s. 2 .

[0128]

[0129] Table 3 Comparison of results with and without LLM-CRaft under different environments.

[0130] In addition, three ablation experiments were conducted under partial LLM-CRaft configurations, with 20 tests performed in each scenario across four environments. The results shown in Table 4 are averaged to evaluate the impact of each component on the effectiveness of the LLM-CRaft mechanism.

[0131] Ablation type crossroads Congested roads highway Lack of inter-group integration collision Congestion collision Lack of in-group selection collision collision Congestion Lack of intra-group synchronization Congestion collision collision

[0132] Table 4 Ablation experimental results for different missing components

[0133] The ablation experiments of this invention focus on two core components: real-time group dynamics and the consensus system. The method of this invention is tested by omitting steps such as group merging, leader selection, and synchronization.

[0134] Table 4 presents the complete ablation study results, indicating that:

[0135] 1) Dynamic grouping significantly improves system performance and reduces the computational load of LLM;

[0136] 2) Omitting the election mechanism will increase the risk of collisions.

[0137] Skipping these steps can lead to collisions, while a complete system can avoid problems, demonstrating that every step in the Raft process is crucial.

[0138] The experiments of this invention show that LLM-CRaft can be effectively expanded as the functionality of LLM is improved. Figure 5 and Figure 6 This demonstrates the extended behavior across multiple models and performance metrics. Figure 5 and Figure 6 The experimental results demonstrating scalability highlight the scalability of LLM-CRaft thanks to LLM intelligence. The evaluation metrics for LLM intelligence primarily focus on model size and the model's performance across various indicators.

[0139] Figure 5 This study reveals a strong correlation between LLM intelligence and driving performance. With improvements in LLM intelligence, the LLM-CRaft method demonstrates a more significant improvement in average speed compared to other methods, while also achieving a greater reduction in collision rate. Similarly, using a more powerful model results in a more optimized acceleration curve, indicating smoother traffic flow and better anticipated behavior.

[0140] Figure 6 The comparative analysis further confirms these extended advantages. LLM-CRaft not only achieved the highest performance score but also exhibited the steepest improvement curve among all models. This superlinear extension relationship indicates that the method of this invention becomes increasingly effective as the model's capabilities improve, resulting in a multiplier effect. When improvements to the base model are combined with the LLM-CRaft framework, they yield disproportionately large gains.

[0141] The performance gap is particularly pronounced in advanced models such as GPT-4-Turbo and Gemini-2, where the method of this invention demonstrates a greater improvement over less complex models. Importantly, even with lower-performing models, the method of this invention consistently maintains a low collision rate, thereby ensuring safety across the entire LLM capability range.

[0142] These results validate the present invention's method of separating coordination logic from individual decision-making, thereby creating a system that can effectively leverage continuous improvements to the underlying model without architectural changes. As LLM continues to improve, the LLM-CRaft framework ensures that these improvements directly translate into better autonomous driving performance.

[0143] This invention proposes the LLM-CRaft framework, an innovative framework that coordinates LLM with distributed consensus, aiming to overcome the gaps in semantic coordination in intelligent transportation systems. By integrating dynamic vehicle grouping, Raft-based data synchronization, and hybrid control, the system of this invention achieves a 20% collision reduction and a 2 m / s speed improvement in congested scenarios compared to an egocentric baseline. Experiments in four different driving environments demonstrate that LLM-CRaft consistently outperforms existing LLM control methods, reducing completion time by 10-30% and significantly improving completion rate.

[0144] While the framework of this invention demonstrates significant advantages, several key limitations exist. The inherent ambiguity of natural language descriptions can occasionally lead to misunderstandings of complex scenarios. While dynamic grouping mechanisms achieve better functionality, they increase V2V communication overhead compared to static cluster configurations, an issue that Raft-based message compression can partially mitigate. Crucially, the experimental validation of this invention remains limited to four standardized driving scenarios in a simulated environment, neglecting key real-world deployment challenges: sensor noise variations (±2.3dB in field testing), unpredictable human-vehicle interactions, and heterogeneous communication latency observed in 5G-V2X prototypes. These gaps between simulation and reality require significant reconciliation efforts across three dimensions: hardware-in-the-loop verification of sensor fusion consistency, secure authentication protocols for hybrid AI control systems, and infrastructure upgrades supporting city-scale semantic consensus.

[0145] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the various method embodiments of this application. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0146] Furthermore, embodiments of this application also provide a system for semantic perception coordination of autonomous vehicles, including a memory for storing computer-executable instructions and a processor; the processor is used to implement the steps in the above-described method embodiments when executing the computer-executable instructions in the memory. The processor may be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Microcontroller Unit (MCU), Neural Processing Unit (NPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices. The aforementioned memory may be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive, etc. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0147] Furthermore, embodiments of this application also provide a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps in the above-described method embodiments.

[0148] All references to this specification are considered to be incorporated integrally into the disclosure of this application so that they can serve as the basis for modifications if necessary. Furthermore, it should be understood that the above descriptions are merely preferred embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

[0149] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for semantic perception coordination of autonomous vehicles, characterized in that, The method comprises: collecting state information of vehicles in real time, and dividing the vehicles into different temporary groups according to spatial position relationships between the vehicles; adopting a Raft protocol for data synchronization and leader election in each group, periodically uploading state information of a follower in each temporary group to a leader, and generating a consistency state matrix of the group by the leader according to the collected state information of the followers in the group; broadcasting the consistency state matrix to all followers in each group by the leader, inputting the consistency state matrix into a large language model, converting the consistency state matrix into a natural language description by the large language model, and generating action parameters of each group member; and inputting the action parameters into a rule-based driving model by each vehicle, generating control instructions according to state information of the vehicle itself by the rule-based driving model, and realizing motion consistency of the vehicles in the group.

2. The method of claim 1, wherein, According to the spatial position relationships between the vehicles, the vehicles are divided into different temporary groups, which further comprise: vehicles with a spatial distance less than or equal to a first preset threshold are divided into a group.

3. The method of claim 2, wherein, Further comprising: when a vehicle is a single group, if the spatial distance between the vehicle and a vehicle in another group at the current time is less than a preset merging threshold, the vehicle is merged into the group; if the spatial distance between a vehicle in a group and any one of the other vehicles in the group at the current time is greater than a preset splitting threshold, the vehicle is split into a single group; wherein the preset merging threshold is less than the first preset threshold, and the first preset threshold is less than the preset splitting threshold.

4. The method of claim 3, wherein, The motion consistency means that the speed directions are basically consistent and the difference between the speeds is less than or equal to a second preset threshold.

5. The method of claim 1, wherein, The action parameters include an acceleration range and a steering angle range.

6. The method of claim 1, wherein, The state information of the vehicle includes position coordinates, speed, acceleration, steering angle, road condition information, and surrounding vehicle state.

7. The method of claim 1, wherein, Further comprising: all vehicles in the group are set as followers, and the state of the current leader is detected through a heartbeat mechanism, and if the heartbeat of the current leader is not detected, leader election is re-performed; if the number of times of leader election failure exceeds a preset number of times, the group is dissolved, and the vehicles in the group are grouped and added to other groups.

8. The method of claim 1, wherein, Further comprising: if the response time of the large language model exceeds a predetermined time, the leader is switched to adopt a proportional-integral-derivative controller.

9. The method of claim 1, wherein, Further comprising: the leader in each group sets different priorities according to different broadcast messages.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer executable instructions, and the computer executable instructions are executed by a processor to implement the steps in the method of any one of claims 1 to 9.

Images