Intersection passing method based on multi-modal language model and vehicle intention recognition

Abstract

The application relates to the technical field of intelligent traffic, in particular to a crossing passing method based on a multi-modal language model and vehicle intention recognition, which comprises the following steps: obtaining vehicle intention messages and roadside perception data of a target crossing, and forming vehicle intention records; obtaining candidate passing priorities or candidate passing windows by using a large model, and performing phase constraint and conflict constraint checking on the candidate passing priorities or the candidate passing windows to obtain a coordinated passing scheme; when a deviation between a monitored vehicle actual trajectory and the vehicle intention records meets a preset condition, corresponding passing windows are cancelled, a coordinated passing scheme is regenerated or a safety fallback scheme is executed. The crossing passing method based on the multi-modal language model and the vehicle intention recognition aims to solve the problems that environment perception is insufficient in a crossing coordinated passing method, a traditional model generation scheme is difficult to meet physical safety constraints, and passing windows are difficult to be cancelled in time when a vehicle actual trajectory deviates.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to an intersection passage method based on multimodal language models and vehicle intent recognition. Background Technology

[0002] Intersections are the areas with the most intensive spatiotemporal interaction between vehicles, non-motorized vehicles, and pedestrians in urban road networks. Especially during rush hours, inclement weather, temporary construction, priority vehicle passage, and tidal traffic flow, control based solely on fixed timing or single perception results struggles to balance traffic efficiency and conflict risks. Vehicle-to-Everything (V2X) communication can provide roadside systems with prior information such as vehicle location, speed, target lane, turning intention, and estimated arrival time, enabling refined collaborative traffic flow at intersections. However, existing communication methods can be affected by communication latency, location drift, message loss, temporary lane changes, driver or autonomous driving system strategy changes, and abnormal messages. If traffic control systems directly use vehicle-reported intentions as the basis for release or reservation, it can easily lead to discrepancies between the actual vehicle trajectory and the allocated passage window, resulting in phase conflicts, duplicate occupation of conflict zones, or insufficient protection for pedestrians and non-motorized vehicles.

[0003] Large models have the ability to comprehensively understand heterogeneous scene information and generate candidate collaborative strategies. However, directly using large models to output phases, green light times, or vehicle release instructions still results in problems such as outputs that do not conform to intersection phase rules, inability to explain the basis of intent, and difficulty in timely handling of vehicle intent deviations.

[0004] Therefore, a method for navigating intersections based on a multimodal language model and vehicle intent recognition is provided. Summary of the Invention

[0005] (1) Technical problems to be solved This invention provides an intersection passage method based on a multimodal language model and vehicle intent recognition, which solves the technical problems of insufficient environmental perception, difficulty in meeting physical safety constraints by traditional model generation schemes, and difficulty in timely cancellation of passage windows when the actual vehicle trajectory deviates from the target path in intersection cooperative passage methods.

[0006] (2) Technical solution This invention provides a method for navigating intersections based on a multimodal language model and vehicle intent recognition, comprising the following steps: Acquire vehicle intent messages and roadside perception data at the target intersection; A vehicle intent record is formed based on the vehicle intent message and the roadside perception data; The vehicle intent record is verified to obtain the intent credibility. A spatiotemporal conflict map is constructed based on vehicle information, lane groups, and predicted trajectories; The structured scene information, the credibility of the intent, and the spatiotemporal conflict graph are input into the large model to obtain candidate passage priorities or candidate passage windows. Phase constraints and conflict constraints are verified on the candidate passage priorities or the candidate passage windows to obtain a cooperative passage scheme; Control commands are sent to the vehicle-mounted terminal according to the aforementioned collaborative traffic scheme; When the deviation between the actual trajectory of the monitored vehicle and the vehicle's intended record meets preset conditions, the corresponding passage window is canceled, a new collaborative passage plan is generated, or a safety rollback plan is executed.

[0007] Furthermore, the vehicle intent message includes at least two of the following: temporary vehicle identifier, message generation time, vehicle position, speed, acceleration, heading angle, target lane identifier, target turning direction, and predicted arrival time in the conflict zone.

[0008] Furthermore, the roadside perception data includes target trajectory, target category, location, speed, heading angle, and lane affiliation information obtained by one or more of cameras, millimeter-wave radar, lidar, or geomagnetic sensors.

[0009] Furthermore, the vehicle intent record is based on the association between the vehicle identifier and the roadside target trajectory based on position distance, speed difference, heading angle difference, and timestamp difference.

[0010] Furthermore, the credibility of the intent is determined at least by the consistency of message timeliness, the consistency of location movement, the consistency of historical behavior fulfillment, and the compatibility of road network phase rules.

[0011] Furthermore, the nodes of the spatiotemporal conflict graph include traffic participants, lane groups, conflict zones, and predicted occupancy time windows, while the edges are used to represent the time overlap relationship between two traffic participants occupying the same conflict zone or the conflict relationship between two lane groups.

[0012] Furthermore, the output information of the large model includes one or more of the following: candidate lane groups, candidate traffic priorities, candidate time windows, candidate phase sequences, and corresponding structured reasons.

[0013] Furthermore, the step of performing phase constraint and conflict constraint verification on the candidate passage priority or the candidate passage window to obtain a cooperative passage scheme specifically involves: Based on the current phase, allowed phase transition relationship, minimum green light time, yellow light time, all-red light time, pedestrian clearance time, non-motorized vehicle protection time, and conflict zone mutual exclusion relationship, the candidate traffic priority or the candidate traffic window is verified; when the candidate result does not meet any constraint, a structured counterexample containing the violated constraint and associated conflict zone is generated, and the structured counterexample is fed back to the large model to generate the cooperative traffic scheme.

[0014] Furthermore, the preset conditions include: at least one of the following: position deviation, heading deviation, arrival time deviation, or inconsistency between actual turning and intended turning, which exceeds a threshold.

[0015] The present invention also provides a system employing the above-described intersection passage method based on multimodal language model and vehicle intent recognition, comprising: The data acquisition module is used to acquire vehicle intent messages and roadside perception data at the target intersection; The intent recording module is used to form a vehicle intent record based on the vehicle intent message and the roadside perception data; The trust verification module is used to perform trust verification on the vehicle intent record to obtain the intent trustworthiness. The conflict construction module is used to build a spatiotemporal conflict map based on vehicle information, lane groups, and predicted trajectories. The collaborative decision-making module is used to input structured scene information, the credibility of the intent, and the spatiotemporal conflict graph into the large model to obtain candidate passage priorities or candidate passage windows; A secure compilation module is used to perform phase constraint and conflict constraint verification on the candidate passage priority or the candidate passage window to obtain a cooperative passage scheme. The control execution module is used to send control commands to the vehicle terminal according to the cooperative passage scheme. The online monitoring module is used to cancel the corresponding passage window, regenerate the collaborative passage plan, or execute a safety rollback plan when the deviation between the actual trajectory of the monitored vehicle and the vehicle's intended record meets preset conditions.

[0016] (3) Beneficial effects In summary, this invention improves the reliability of traffic window allocation by jointly verifying vehicle-reported intentions with roadside measured trajectories, historical compliance behavior, and road network phase rules, avoiding the direct use of unreliable or distorted vehicle turning intentions as the basis for coordinated passage. By limiting the large model to generating structured candidate priorities and candidate windows and completing phase and conflict constraint verification, the large model's scenario understanding capability and intersection physical safety constraints form a closed loop, reducing the probability of issuing unexecutable or conflicting control schemes. Simultaneously, by detecting deviations between intentions and actual trajectories online and executing cancellation, reassignment, and safe backoff, it can adapt to temporary lane changes, sudden turns, communication quality degradation, and abnormal events, improving the robustness and interpretability of complex intersections under mixed traffic conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an intersection passage method based on a multimodal language model and vehicle intent recognition provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the multimodal language model training process; Figure 3 This is a schematic diagram of the structure of an intersection traffic system based on a multimodal language model and vehicle intent recognition provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of vehicle intent credibility verification and passage window allocation provided in Embodiment 1 of the present invention.

[0019] In the picture: 201-Data Acquisition Module; 202-Intent Recording Module; 203-Trust Verification Module; 204-Conflict Construction Module; 205-Collaborative Decision-Making Module; 206-Secure Compilation Module; 207-Control Execution Module; 208-Online Monitoring Module. Detailed Implementation

[0020] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention, that is, the present invention is not limited to the described embodiments.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] A first aspect of this invention provides an intersection passage method based on a multimodal language model and vehicle intent recognition, see [link to relevant documentation]. Figure 1 The method includes the following steps: S101. Obtain vehicle intent messages and roadside perception data at the target intersection.

[0023] Specifically, the system collects V2X intent messages, video images, millimeter-wave radar or lidar data points, intersection MAP (map) information, SPaT (Signal Phase and Timing) information, and the current status of the signal controller at the target intersection. MAP information describes the relationships between approach lanes, exit lanes, lane groups, stop lines, conflict zones, and achievable turns; SPaT information describes the current phase, remaining phase time, and preset phase transition relationships.

[0024] The vehicle intent message includes at least two of the following: temporary vehicle identification, message generation time, vehicle position, speed, acceleration, heading angle, target lane identification, target turning direction, and predicted arrival time at the conflict zone. Roadside perception data includes target trajectory, target category, position, speed, heading angle, and lane assignment information obtained from one or more of cameras, millimeter-wave radar, lidar, or geomagnetic sensors.

[0025] S102. Form a vehicle intent record based on the vehicle intent message and roadside perception data.

[0026] Specifically, data from sources such as video, radar, and trajectory are synchronized in time and unified in coordinates, and temporary identifiers of connected vehicles are associated with roadside target trajectories. Each intent record includes at least the vehicle identifier, target lane, whether turning / going straight, predicted entry time into the conflict zone, predicted exit time from the conflict zone, message timestamp, and associated roadside trajectory identifier. For messages that cannot be reliably associated with roadside trajectories, their message state is retained but not directly used for high-priority window allocation.

[0027] Among them, vehicle intent recording associates vehicle identifiers with roadside target trajectories based on location distance, speed difference, heading angle difference, and timestamp difference.

[0028] S103. Perform a reliability check on the vehicle intent record to obtain the intent reliability.

[0029] Specifically, message timeliness consistency, location and motion consistency, historical behavior fulfillment consistency, and road network phase rule compatibility are calculated separately. Message timeliness consistency reflects message age, duration of consecutive packet loss, and timestamp jumps; location and motion consistency reflects the spatial and kinematic deviations between the vehicle's reported trajectory and the roadside fused trajectory; historical behavior fulfillment consistency reflects the degree of conformity between the vehicle's reported intention and the actual turning, lane changing, and arrival time within the sliding time window; and road network phase rule compatibility determines whether the target lane, target turn, and predicted conflict zone are compatible with MAP information, current phase, and allowed phase transition relationships.

[0030] Optionally, for the i-th vehicle, the aforementioned message timeliness consistency, location motion consistency, historical behavior fulfillment consistency, and road network phase rule compatibility are denoted as c_time[i], c_motion[i], c_behavior[i], and c_rule[i], respectively, and the credibility C_i is determined according to preset weights. The weights can be adjusted based on the availability of intersection sensors, weather, communication quality, or vehicle category. The system presets the weights to determine the credibility, mapping C_i to high, medium, or low credibility levels, and simultaneously saves each consistency quantity and evidence items of anomalies for subsequent decision-making and manual review.

[0031] S104. Construct a spatiotemporal conflict map based on vehicle information, lane groups, and predicted trajectories.

[0032] Specifically, based on the reachability of lane groups and the predicted trajectories of traffic participants, vehicles, pedestrians, and non-motorized vehicles are mapped to one or more conflict zones, and a predicted occupancy time window is generated for each conflict zone. For two nodes occupying the same conflict zone and whose time windows overlap, a conflict edge is established in the graph; for nodes in mutually conflicting lane groups, a conflict edge is also established. Low-confidence vehicles form a conservative predicted occupancy time window according to the roadside fusion trajectory.

[0033] S105. Input the structured scene information, intent credibility, and spatiotemporal conflict graph into the large model to obtain candidate access priorities or candidate access windows.

[0034] Specifically, in the large-scale model collaborative decision-making stage, traffic participant states, vehicle intent credibility, conflict graph summaries, MAP / SPaT states, priority vehicle requests, and current safety constraints are encoded as structured scenario inputs. The large-scale model outputs one or more of the following: candidate lane groups, candidate passage priorities, candidate time windows, and candidate phase sequences, without directly generating controller execution messages that can bypass safety constraints. Optionally, using historical samples containing true intents, deviated intents, and corresponding handling results, the large-scale model is fine-tuned under supervision using LoRA (Low-Rank Adaptation), enabling it to learn to reduce the collaborative priority of corresponding vehicles when intent credibility decreases.

[0035] The output information of the large model includes one or more of the following: candidate lane groups, candidate traffic priorities, candidate time windows, candidate phase sequences, and corresponding structured reasons.

[0036] like Figure 2 During the training of the multimodal language model, the LoRA fine-tuning module is used to perform supervised fine-tuning by representing each training example as an image, instruction, or standard answer. The basic idea of ​​LoRA is to use two low-rank matrices to approximate the change in weights, rather than directly learning the change in the entire weight matrix. For a linear layer, its initial weight matrix is ​​denoted as... The input vector is So the original output of this linear layer is: This is the low-rank weight update matrix introduced by LoRA: In the formula, and It is a low-rank matrix. It is a rank parameter, and Much smaller than the input and output dimensions, their relationship can be expressed as: ; Introducing scaling factor Then, LoRA's output h It can be written as: During training, initial weights Fixed, only optimize low-rank factors and If the original linear layer had With one parameter, LoRA only increases by approximately There are trainable parameters. And when the rank... When the input-output dimensions are very small, the number of trainable parameters required is far less than the number of parameters needed to fine-tune the entire model. Therefore, LoRA can be tuned for downstream tasks with very few additional parameters and saves GPU memory and storage space.

[0037] Use autoregressive supervised fine-tuning of the target to refine traffic scene images. Task instructions and the target output sequence This is placed within a given generation task. Assume the target sequence is... Therefore, the probability that the model generates the correct answer given the image and instructions is: ; The corresponding training loss is expressed in the form of cross-entropy: ; In the formula, The length of the target answer sequence. For the first One target byte, For all the bytes preceding it, I and T represent the input image and task instructions, respectively. is the cross-entropy loss function used for training. This loss function allows the model to generate the correct sequence of target answers as much as possible, given an image and an instruction.

[0038] LoRA fine-tuning uses a smaller rank parameter to reduce fine-tuning costs and uses linear learning rate decay, gradient accumulation, and half-precision training to improve training robustness.

[0039] S106. Perform phase constraint and conflict constraint verification on candidate passage priorities or candidate passage windows to obtain a cooperative passage scheme.

[0040] Specifically, a safety compiler maps candidate results to phase schemes that the phase controller can execute and / or suggested passage windows for the vehicle terminal. The safety compiler verifies at least the phase reachability, phase duration, yellow and all-red light times, pedestrian and non-motorized vehicle clearance times, conflict zone mutual exclusion relationships, and priority vehicle protection conditions of the candidate schemes. When a candidate scheme does not meet the constraints, the safety compiler does not issue the scheme but instead generates a structured counterexample. This structured counterexample includes the violated constraints, associated conflict zones, relevant vehicle identifiers, and time intervals, and based on this, requests the large model to generate a corrected candidate scheme (i.e., a cooperative passage scheme).

[0041] S107. Issue control commands to the vehicle terminal according to the collaborative passage scheme.

[0042] S108. When the deviation between the actual trajectory of the monitored vehicle and the vehicle's intended record meets the preset conditions, cancel the corresponding passage window, regenerate the collaborative passage plan, or execute the safety rollback plan.

[0043] Specifically, the actual vehicle trajectory is continuously compared with the high- or medium-confidence intent records. When the actual turn, target lane, arrival time, position, or heading deviates from the intent prediction by more than a threshold, the vehicle's degree is reduced and its unexecuted passage window is revoked; without violating the existing safety constraints of other participants, the conflict map is reconstructed, windows are reallocated, or the system switches to a preset safe timing.

[0044] The preset conditions include: at least one of the following: the positional deviation, heading deviation, arrival time deviation, or inconsistency between the actual turn and the intended turn of the actual trajectory relative to the predicted trajectory exceeds a threshold.

[0045] A second aspect of this invention provides a system employing the aforementioned intersection passage method based on a multimodal language model and vehicle intent recognition, see [link to relevant documentation]. Figure 3 The system may include: The data acquisition module 201 is used to acquire vehicle intent messages and roadside perception data at the target intersection; Intent recording module 202 is used to form vehicle intent records based on vehicle intent messages and roadside perception data; The trust verification module 203 is used to perform trust verification on the vehicle intent record to obtain the intent trustworthiness. Conflict construction module 204 is used to construct a spatiotemporal conflict map based on vehicle information, lane groups and predicted trajectories; The collaborative decision-making module 205 is used to input structured scene information, intent credibility, and spatiotemporal conflict map into the large model to obtain candidate access priorities or candidate access windows. The secure compilation module 206 is used to perform phase constraint and conflict constraint verification on candidate passage priorities or candidate passage windows to obtain a cooperative passage scheme. The control execution module 207 is used to send control commands to the vehicle terminal according to the cooperative passage scheme; The online monitoring module 208 is used to cancel the corresponding passage window, regenerate the cooperative passage plan, or execute the safety rollback plan when the deviation between the actual trajectory of the monitored vehicle and the vehicle's intention record meets preset conditions.

[0046] Example 1 A vehicle reporting a left-turn intention reports that it is located in the left-turn lane at the east entrance, plans to turn left, and will enter the central conflict zone at a certain time. The system detects that the vehicle's position, speed, and heading are consistent with the roadside trajectory. Historically, the reported left-turn intentions and actual left-turn behaviors have a high degree of consistency. Based on the existing vehicle trajectories, the system constructs a spatiotemporal conflict map. After the large model reads the structured coded intersection status, it can output candidate solutions. For example... Figure 4As shown, the large model generates candidate solutions. The safety compiler then checks the relationship between the current vehicle's candidate window and the straight-ahead traffic flow, pedestrian crossing phase, yellow light time, and all-red light time. Only when the safety constraints are met is the solution converted into the next phase solution and a suggested passage window sent to the current vehicle. If a vehicle changes lanes before the passage window takes effect, or if its actual heading deviates from the predicted heading by a set threshold, the online monitoring module generates a deviation event, cancels the passage window that the current vehicle has not yet executed, and reduces the current vehicle's confidence level to medium or low confidence. The system regenerates candidate solutions based on the new spatiotemporal conflict map; if a candidate solution that meets the safety constraints cannot be obtained within a specified time, the system switches to the pre-configured passage solution.

[0047] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. The present invention is not limited to the specific steps and structures described above and shown in the figures. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0048] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art without departing from the scope of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for navigating intersections based on a multimodal language model and vehicle intent recognition, characterized in that, The method includes the following steps: Acquire vehicle intent messages and roadside perception data at the target intersection; A vehicle intent record is formed based on the vehicle intent message and the roadside perception data; The vehicle intent record is verified to obtain the intent credibility. A spatiotemporal conflict map is constructed based on vehicle information, lane groups, and predicted trajectories; The structured scene information, the credibility of the intent, and the spatiotemporal conflict graph are input into the large model to obtain candidate passage priorities or candidate passage windows. Phase constraints and conflict constraints are verified on the candidate passage priorities or the candidate passage windows to obtain a cooperative passage scheme; Control commands are sent to the vehicle-mounted terminal according to the aforementioned collaborative traffic scheme; When the deviation between the actual trajectory of the monitored vehicle and the vehicle's intended record meets preset conditions, the corresponding passage window is canceled, a new collaborative passage plan is generated, or a safety rollback plan is executed.

2. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The vehicle intent message includes at least two of the following: temporary vehicle identifier, message generation time, vehicle position, speed, acceleration, heading angle, target lane identifier, target turning direction, and predicted arrival time in the conflict zone.

3. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The roadside perception data includes target trajectory, target category, location, speed, heading angle, and lane affiliation information obtained by one or more of cameras, millimeter-wave radar, lidar, or geomagnetic sensors.

4. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The vehicle intent record is based on the association between vehicle identifiers and roadside target trajectories using location distance, speed difference, heading angle difference, and timestamp difference.

5. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The credibility of the intent is determined at least by the consistency of message timeliness, the consistency of location movement, the consistency of historical behavior fulfillment, and the compatibility of road network phase rules.

6. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The nodes of the spatiotemporal conflict graph include traffic participants, lane groups, conflict zones, and predicted occupancy time windows. The edges are used to represent the time overlap relationship between two traffic participants occupying the same conflict zone or the conflict relationship between two lane groups.

7. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The output information of the large model includes one or more of the following: candidate lane groups, candidate traffic priorities, candidate time windows, candidate phase sequences, and corresponding structured reasons.

8. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The step of performing phase constraint and conflict constraint verification on the candidate passage priority or the candidate passage window to obtain a cooperative passage scheme is as follows: Based on the current phase, allowed phase transition relationship, minimum green light time, yellow light time, all-red light time, pedestrian clearance time, non-motorized vehicle protection time, and conflict zone mutual exclusion relationship, the candidate traffic priority or the candidate traffic window is verified; when the candidate result does not meet any constraint, a structured counterexample containing the violated constraint and associated conflict zone is generated, and the structured counterexample is fed back to the large model to generate the cooperative traffic scheme.

9. The intersection passage method based on multimodal language model and vehicle intent recognition according to claim 1, characterized in that, The preset conditions include: at least one of the following: position deviation, heading deviation, arrival time deviation, or inconsistency between actual turn and intended turn, which exceeds a threshold.

Images