Intelligent driving cooperative control terminal based on visual language action model

Abstract

The application discloses an intelligent driving cooperative control terminal based on a visual language action model, comprising an instruction optimization processing unit, an execution management analysis unit and a dynamic self-adaptive cooperative processing unit, and relates to the technical field of intelligent cooperative control, solves the technical problem of how to realize strategy dynamic switching based on real-time scene classification and improve the self-adaptive ability of a system to a complex traffic scene, and the application carries out conflict analysis on continuous actions after disassembling the continuous actions through the dynamic self-adaptive cooperative processing unit, clearly defines the grading standard of emergency actions and non-emergency actions, adopts a pause-recovery mechanism for non-emergency actions, resolves conflicts through parameter adjustment or equivalent safety substitution for emergency actions, improves the continuity and safety of action execution, realizes scene fine-grained classification based on a multi-modal fusion model, triggers strategy switching through scene confidence, and starts the corresponding strategy after all sub-scene confidences of a composite scene reach the standard.

Description

Technical Field

[0001] This invention relates to the field of intelligent collaborative control technology, specifically to an intelligent driving collaborative control terminal based on a visual language action model. Background Technology

[0002] With the development of intelligent driving technology, multi-sensor fusion and human-machine interaction have become the core foundation for achieving collaborative control. In intelligent driving systems, perception, decision-making, and control are the key technologies.

[0003] According to patent application CN120348312A, a cloud-edge collaborative intelligent driving control method and system are disclosed. This method utilizes Attention-GAN to optimize cloud-vehicle collaborative behavioral decisions. The framework first leverages an attention mechanism to enable the model to adaptively focus on key information in the environment, and then introduces GAN to optimize driving strategies, improving the accuracy and robustness of decisions. Based on this, a cloud-edge heterogeneous collaborative system architecture is built, achieving distributed collaborative optimization through federated learning. This leverages the powerful computing capabilities of the cloud to learn general patterns from massive amounts of data, while also utilizing the real-time processing capabilities of the vehicle for personalized decisions. Furthermore, strategies such as incremental model updates and selective communication uploads are designed to save communication overhead while protecting user privacy.

[0004] However, existing intelligent driving systems need to handle multiple tasks such as environmental perception, command parsing, and action execution in complex traffic scenarios, but they have the following limitations:

[0005] The command execution condition verification is singular and relies on a single dimension, lacking a comprehensive judgment on environmental adaptability and vehicle status, which easily leads to the problem that the command is feasible but the scenario is not suitable.

[0006] The conflict resolution mechanism for consecutive actions is imperfect, and the classification strategy for emergency and non-emergency actions is not clearly defined, which can easily lead to confusion in execution due to conflicting actions.

[0007] It has weak adaptive capabilities for dynamic scenarios, and the strategy switching for scenarios such as urban roads, highways, and severe weather relies on manual presets, lacking a dynamic triggering mechanism based on real-time scenario confidence. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides an intelligent driving cooperative control terminal based on a visual language action model, which solves the problem of how to achieve dynamic switching of strategies based on real-time scene classification and improve the system's adaptability to complex traffic scenarios.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an intelligent driving cooperative control terminal based on a visual language action model, comprising:

[0010] The instruction optimization processing unit is used to optimize voice instructions, generate structured instructions, and transmit them to the execution management and analysis unit.

[0011] The execution management and analysis unit is used to analyze structured instructions, obtain the real-time environment and analyze the instruction execution conditions. If the conditions are met, an execution action is generated; if not, the specific reasons are determined, an execution action is generated and transmitted to the dynamic adaptive collaborative processing unit.

[0012] The dynamic adaptive collaborative processing unit is used to receive execution actions and perform multi-objective collaborative analysis and environmental perception collaborative analysis. The multi-objective collaborative analysis method is to decompose the execution actions into coherent actions. If there are conflicts, the conflicting actions are classified and processed separately. If there are no conflicts, the execution efficiency is improved by compressing invalid waiting time and optimizing hardware resource allocation, and multi-objective collaborative processing information is generated.

[0013] The environmental perception collaborative analysis method involves classifying scenarios, triggering strategy switching based on scenario classification results and risk indicators, and forcibly switching to a safety fallback strategy when risk indicators exceed the limit, thereby generating environmental perception collaborative processing information.

[0014] As a further aspect of the present invention, a sensor data acquisition unit is included, which is used to acquire sensor data from a camera, lidar, millimeter-wave radar and microphone, and obtain voice commands, and transmit the sensor data and voice commands to the command optimization processing unit.

[0015] The collaborative control information output unit is used to collaboratively process and execute multi-target collaborative processing information and environmental perception collaborative processing information.

[0016] As a further aspect of the present invention, the optimization process includes:

[0017] The system employs a microphone array combined with beamforming technology to directionally acquire driver speech and suppress external noise, and then uses a deep learning noise reduction model for noise reduction processing. It distinguishes between valid commands and noise through voice activity detection, binds environmental visual features, vehicle motion parameters and positioning information to each command, and constructs command-scene pairing data. Based on acoustic models and pre-trained language models in the driving domain, it parses the core intent of commands, disambiguates ambiguous commands, and outputs structured commands.

[0018] As a further aspect of the present invention, the structured instructions are analyzed to identify the vehicle control module and key parameters corresponding to the instructions; a three-dimensional verification system of safety-feasibility-scenario adaptation is constructed based on multi-source real-time environmental data to determine whether the execution conditions are met; if met, the structured instructions are converted into parameterized execution signals; if not met, the unmet factors are located through root cause analysis, and corresponding execution actions are generated.

[0019] As a further aspect of the present invention, a three-dimensional verification system for safety, feasibility, and scenario adaptation is constructed based on multi-source real-time environmental data, including safety constraint verification, which includes verification of collision time (TTC), minimum safe distance, and traffic rule compliance.

[0020] Feasibility verification includes verifying the vehicle's power, braking, and steering system capabilities as well as its adaptability to road conditions.

[0021] Scene adaptation verification includes verifying the matching of instructions with the current road type and traffic flow status.

[0022] As a further aspect of the present invention, the conflicting actions are classified and processed separately. Consecutive actions with conflicts are recorded as conflicting actions, and the conflict type of the conflicting actions is obtained. At the same time, non-urgent actions and urgent actions are classified. For non-urgent actions, execution is suspended and resumed after the conflict is resolved. For urgent actions, parameters are adjusted to avoid conflicts or to replace conflicting actions with equivalent safe actions, and conflict coordination information is generated.

[0023] As a further aspect of the present invention, if there is no conflict, the sequential action time window is analyzed, the dependency relationship is identified by using a time-series alignment algorithm, rigid constraints and flexible windows are marked, and the redundancy waiting of flexible windows is compressed based on the real-time environment; at the same time, the occupation of hardware resources such as computing power, sensors, and actuators by the action is analyzed, resource requirements are marked according to demand, and a demand-based allocation and priority preemption mechanism is adopted to generate normal collaborative processing information.

[0024] As a further aspect of the present invention, the environmental perception collaborative analysis method is as follows: classify the scene, automatically switch the collaborative strategy for different scenes, obtain the real-time scene, determine the corresponding scene classification result, and then set the trigger condition based on the scene classification result. The trigger condition includes the scene confidence level. When the classification confidence level of the target scene is greater than or equal to the switching threshold, the strategy conversion is initiated. For composite scenes, the corresponding composite strategy is triggered only when the confidence levels of all scenes meet the standard.

[0025] As a further aspect of the present invention, the generation of environmental perception collaborative processing information includes:

[0026] The system monitors risk indicators based on the current scenario classification results. These indicators include following safety distance, lateral intrusion risk, and intersection conflict risk. The system compares these indicators with risk thresholds, the specific values ​​of which are set by the operator. When a risk indicator exceeds the risk threshold, the system forcibly switches to a safety fallback strategy, which has higher priority than the regular scenario strategy. Simultaneously, the system monitors the risk indicator in real time. After three consecutive frames, the system returns the risk indicator to the safe range and gradually recovers from emergency braking to regular control. Once the transition is complete, the system automatically switches back to the original scenario strategy and generates environmental perception collaborative processing information.

[0027] As a further aspect of the present invention, the safety fallback strategy includes:

[0028] Vehicle control: Trigger ABS+ESP coordinated braking to reduce the vehicle speed to a safe value or bring it to a stop, and simultaneously make a small lateral turn when there is space.

[0029] Full-area warning: Activate hazard lights, honk the horn, flash the high-mounted brake light, and provide external voice announcements;

[0030] System Lockdown: Freeze non-safe functions, allocate all computing power and executor permissions to risk avoidance control, and smoothly restore to the original strategy after the risk is eliminated.

[0031] This invention provides an intelligent driving cooperative control terminal based on a visual language action model. Compared with existing technologies, it has the following advantages:

[0032] This invention improves driver's voice gain and suppresses external noise by employing an 8-microphone ring array combined with MVDR adaptive beamforming technology; it maintains a high recognition rate even in extremely noisy scenarios by combining wavelet transform and Transformer hybrid noise reduction model; and it establishes a three-dimensional verification system of safety, feasibility, and scenario adaptation. Through comprehensive analysis of multi-source environmental data, vehicle status, and scenario characteristics, it ensures that command execution meets safety constraints and adapts to vehicle capabilities and real-time scenarios, avoiding the risk of blind execution.

[0033] This invention uses a dynamic adaptive collaborative processing unit to break down continuous actions and perform conflict analysis, clarifying the classification criteria for emergency and non-emergency actions. For non-emergency actions, a pause-resume mechanism is adopted, while for emergency actions, conflicts are resolved through parameter adjustment or equivalent safety substitution, thereby improving the continuity and safety of action execution. Based on a multimodal fusion model, fine-grained scene classification is achieved, and strategy switching is triggered by scene confidence. For composite scenes, the corresponding strategy is activated only after the confidence of all sub-scenes reaches the standard. Attached Figure Description

[0034] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1

[0037] Please see Figure 1This application provides an intelligent driving cooperative control terminal based on a visual language action model, including a sensor data acquisition unit, an instruction optimization and processing unit, an execution management and analysis unit, a dynamic adaptive cooperative processing unit, and a cooperative control information output unit. As can be seen from the accompanying drawings, the information between the above functional units is transmitted in one direction.

[0038] The sensor data acquisition unit is used to acquire sensor data from the camera, lidar, millimeter-wave radar, and microphone, and obtain corresponding voice commands, which are then transmitted to the command optimization and processing unit.

[0039] The instruction optimization processing unit is used to optimize the acquired voice instructions, and the specific optimization processing method is as follows:

[0040] The system employs microphone array and beamforming technology to directionally acquire driver voice commands, suppress external noise, and combines a deep learning-based noise reduction model to process the voice commands. It distinguishes between valid commands and noise through voice activity detection, binds visual scene features to each command, constructs command-scene pairing data, and structures the obtained voice commands before transmitting them to the execution management and analysis unit.

[0041] An 8-microphone ring array combined with adaptive beamforming technology, such as the MVDR algorithm, is used to locate the driver's position through sound source localization and form a spatially directional sound pickup beam, which improves the voice signal gain in the driver's seat by more than 15dB, while suppressing external environmental noise by ≥25dB. For extreme noise scenarios, a hybrid noise reduction model based on wavelet transform and Transformer is introduced.

[0042] Real-time capture of environmental visual features, vehicle motion parameters, and positioning information at the moment the command is triggered. Specifically, environmental visual features include lane lines, traffic lights, and obstacle detection results from the forward-facing camera, while vehicle motion parameters include speed, gear, and turn signal status. The above information is packaged with the voice command into command-scene pairing data and stored in a dynamic database through time stamp alignment.

[0043] The acoustic model is used to extract voice command features, and combined with CTC-Attention hybrid decoding, basic speech-to-text conversion is achieved. The language model introduces the BERT model pre-trained in the driving domain to parse the core intent in the command. By searching the command-scene pairing database, ambiguous commands are disambiguated, and finally the structured command is output and transmitted to the execution management analysis unit.

[0044] The execution management analysis unit analyzes the acquired structured instructions, then acquires the current real-time environment and analyzes the execution conditions of the structured instructions. It determines whether the current real-time environment directly meets the execution conditions of the structured instructions. If it does, the structured instructions are executed as the standard, and the execution action is generated and transmitted to the coordination control information output unit. Otherwise, if the conditions are not met, the corresponding non-compliance conditions of the current real-time environment are analyzed, the specific reasons are determined, the real-time environment is monitored, and the execution action is generated based on the real-time environment meeting the execution conditions. Then, the action is transmitted to the coordination control information output unit.

[0045] Identify the vehicle control module corresponding to the instruction, extract key parameters, and synchronously integrate multi-source real-time environmental data to construct a three-dimensional verification system of safety, feasibility, and scenario adaptation. This system determines whether the current environment meets the instruction execution conditions and specifically performs safety constraint verification, feasibility verification, and scenario adaptation verification. Based on the verification results, execution strategies are generated for different scenarios.

[0046] When the conditions are met, the structured instructions are converted into parameterized signals that the control module can directly execute, the instruction ID, execution time, and environmental parameters are recorded, and the execution action is generated and transmitted to the collaborative control information output unit. When the conditions are not met, the root cause analysis model is used to locate the core factors that are not met, which are divided into three categories: environmental conditions that are not yet up to standard, vehicle status restrictions, and safety risks that have not been eliminated, and real-time monitoring is performed.

[0047] For scenarios where the environment does not meet the standards, real-time monitoring is initiated, a preset trigger threshold is set, and transitional actions are generated, such as maintaining the current lane and decelerating to 30km / h while waiting.

[0048] For scenarios with vehicle status restrictions, the constraint information is fed back to the human-machine interaction unit, and alternative solutions are provided.

[0049] For scenarios where safety risks have not been eliminated, an active risk avoidance priority mechanism is triggered, such as maintaining a safe distance from vehicles and continuously monitoring pedestrian dynamics. Once the risk is eliminated, the original instructions are automatically executed.

[0050] The collaborative control information output unit is used to execute actions based on the generated actions.

[0051] Example 2

[0052] As a second embodiment of the present invention, it is implemented based on the first embodiment, and the difference from the first embodiment is as follows:

[0053] The execution management and analysis unit transmits the generated execution actions to the dynamic adaptive collaborative processing unit for processing. Specifically, it performs dynamic adaptive processing based on the obtained execution actions, including multi-objective collaborative analysis and environmental perception collaborative analysis.

[0054] The specific method for multi-target collaborative analysis is to obtain the execution actions and decompose them into coherent actions, and in this case, the decomposition is carried out in the order of execution. At the same time, conflict analysis is performed on the obtained coherent actions. If there is a conflict in the coherent actions, a conflict analysis signal is generated; otherwise, if there is no conflict in the coherent actions, a normal analysis signal is generated.

[0055] The generated conflict analysis signals are processed to obtain the consecutive actions with conflicts and record them as conflict actions. The conflict type of the conflict actions is obtained and classified into non-urgent actions and urgent actions. For non-urgent actions, execution is suspended and resumed after the conflict is resolved. For urgent actions, parameters are adjusted to avoid conflicts or replace conflict actions with equivalent safe actions, and conflict coordination information is generated.

[0056] Based on the level of urgency, actions that are directly related to core security modules and will pose a security risk within 3 seconds if not handled are classified as emergency actions. Actions whose impact is limited to comfort / efficiency functions, or whose conflict mitigation window is ≥5 seconds, are classified as non-emergency actions.

[0057] For non-urgent actions, a pause marker is sent to the execution module, which includes the action ID, current progress, pause timestamp, and target parameters. Low-priority transition actions are started synchronously, and the status of conflict sources is monitored in real time. When the conflict resolution conditions are met, for urgent actions, vertical, horizontal, or temporal parameters are dynamically adjusted. For example, vertical parameters such as adjusting speed / acceleration, horizontal parameters such as optimizing trajectory / angle, and temporal parameters such as delaying action start time. If parameter adjustment cannot completely avoid conflict, an equivalent and safer alternative action is triggered, prioritizing the preservation of the core intent, ensuring that the alternative action is functionally equivalent to the original action, and generating conflict coordination information.

[0058] For the generated normal analysis signals, the execution efficiency of the action sequence is improved by parameter tuning. Specifically, the time window of the execution of consecutive actions is analyzed and the invalid waiting time is compressed. Specifically, the dependency relationship between consecutive actions is identified by the time sequence alignment algorithm, and the rigid time constraints and flexible time windows of each consecutive action are marked. For the flexible time window, redundant waiting is dynamically compressed based on real-time environmental data.

[0059] Simultaneously, the system analyzes the consumption of vehicle hardware resources, such as computing power, sensors, and actuators, in a continuous sequence of actions to avoid resource idleness or over-allocation. Specifically, resource requirements are marked for different continuous actions, including computing power requirements, sensor requirements, and actuator requirements. On-demand allocation and priority preemption are adopted for processing. For urgent actions, computing power is temporarily preempted, while for non-urgent actions, low-priority computing power is allocated to generate normal collaborative processing information.

[0060] The information from the integrated conflict analysis and processing and the normal analysis and processing is used to generate multi-objective collaborative processing information, which is then transmitted to the collaborative control information output unit.

[0061] The specific method for environmental perception collaborative analysis is as follows: scenarios are classified, such as urban roads, highways, severe weather, and congested road sections. Collaborative strategies are automatically switched for different scenarios to obtain real-time scenarios and determine their corresponding scenario classification results. Then, triggering conditions are set based on the scenario classification results. The triggering conditions include scenario confidence and risk status. When the classification confidence of the target scenario is greater than or equal to the switching threshold, the strategy conversion is initiated. For example, if the confidence of an urban intersection increases from 70% to 90%, the switching threshold is 85%. If the 85% threshold is exceeded, the strategy of going straight on the main road is triggered to switch to the strategy of turning at the intersection. For composite scenarios, the confidence of all sub-scenarios must meet the standard before the corresponding composite strategy is triggered.

[0062] Specifically, in highway (smooth / clear weather) scenarios, regarding safety constraints, the following distance must be maintained at more than 1.5 seconds, and the minimum safe distance must be no less than 50 meters; regarding efficiency targets, the cruising speed should be set to the speed limit of the road segment, and the overtaking response time should be controlled within 1 second; the actuator control details are that the acceleration amplitude should not exceed 1.2m / s², the steering angular velocity should not exceed 5° / s, and the braking delay should not exceed 200ms.

[0063] In urban intersections (signal-controlled / congested) scenarios, safety constraints require pedestrians to have a collision time (TTC) of no less than 3 seconds and to slow down 5 meters before the stop line at the intersection; the efficiency target is a green light passage efficiency of no less than 80%, while emphasizing not to rush through, balancing passage efficiency and rule compliance; in terms of actuator control, the starting acceleration should not exceed 0.8 m / s², and the turning radius should not be less than 15 meters.

[0064] For severe weather scenarios (heavy rain / high-speed driving), safety constraints are more stringent. The following distance must be no less than 3 seconds, and the maximum speed is limited to 60 km / h. The efficiency target is that the traffic efficiency should not be less than 50% of the speed limit of the road section, and traffic capacity should be guaranteed as much as possible under the premise of safety priority. The actuator control parameters are that the brake pressure change rate should not exceed 0.3 bar / s to prevent slippage, and the steering angle is limited to within 10°.

[0065] In the construction area (urban road) scenario, the safety constraints require that the distance from the construction area be no less than 3 meters and the vehicle speed not exceed 30km / h; the efficiency target is that the detour time should not exceed 1.2 times that of the original route, reducing the loss of travel time while safely avoiding the construction area; in terms of actuator control, the interval between continuous lane changes should be no less than 3 seconds, and the turn signal should be turned on more than 5 seconds in advance.

[0066] Simultaneously, the risk indicators of the current scenario classification results are monitored. These indicators include following safety distance, lateral intrusion risk, and intersection conflict risk. These are compared with risk thresholds, the specific values ​​of which are set by the operator. When a risk indicator exceeds the threshold, a forced switch to a safety fallback strategy is initiated, prioritizing it over the regular scenario strategy. Specific safety fallback strategies include: vehicle control, triggering ABS+ESP coordinated braking to reduce the vehicle to a safe speed or bring it to a complete stop within the shortest distance; if there is lateral avoidance space, simultaneously executing a small emergency turn, but prioritizing longitudinal braking; full-area warning, activating high-mounted brake light flashing and external voice announcements in addition to hazard warnings and hazard hazard warnings; system lock, temporarily freezing non-safe functions and allocating all computing power and actuator permissions to hazard avoidance control until the risk is eliminated. Real-time monitoring is also performed, with three consecutive frames monitoring the risk indicators to return to the safe range. The system gradually recovers from emergency braking to regular control. After the transition is complete, it automatically switches back to the original scenario strategy, generates environmental perception and collaborative processing information, and transmits it to the collaborative control information output unit.

[0067] The collaborative control information output unit is used to perform corresponding collaborative processing on the acquired multi-target collaborative processing information and environmental perception collaborative processing information.

[0068] Example 3

[0069] As a third embodiment of the present invention, the focus is on combining the implementation processes of the first and second embodiments.

[0070] The data in the above formulas are all calculated using numerical values, without substituting the units of the parameters. In addition, the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0071] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. An intelligent driving cooperative control terminal based on a visual language action model, characterized in that, include: The instruction optimization processing unit is used to optimize voice instructions, generate structured instructions, and transmit them to the execution management and analysis unit. The execution management and analysis unit is used to analyze structured instructions, obtain the real-time environment and analyze the instruction execution conditions. If the conditions are met, an execution action is generated; if not, the specific reasons are determined, an execution action is generated and transmitted to the dynamic adaptive collaborative processing unit. The dynamic adaptive collaborative processing unit is used to receive execution actions and perform multi-objective collaborative analysis and environmental perception collaborative analysis. The multi-objective collaborative analysis method is to decompose the execution actions into coherent actions. If there are conflicts, the conflicting actions are classified and processed separately. If there are no conflicts, the execution efficiency is improved by compressing invalid waiting time and optimizing hardware resource allocation, and multi-objective collaborative processing information is generated. The environmental perception collaborative analysis method involves classifying scenarios, triggering strategy switching based on scenario classification results and risk indicators, and forcibly switching to a safety fallback strategy when risk indicators exceed the limit, thereby generating environmental perception collaborative processing information.

2. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, It also includes a sensor data acquisition unit, which is used to acquire sensor data from cameras, lidar, millimeter-wave radar and microphones, and obtain voice commands, and transmit the sensor data and voice commands to the command optimization and processing unit. The collaborative control information output unit is used to collaboratively process and execute multi-target collaborative processing information and environmental perception collaborative processing information.

3. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, The optimization process includes: The system employs a microphone array combined with beamforming technology to directionally acquire driver speech and suppress external noise, and then uses a deep learning noise reduction model for noise reduction processing. It distinguishes between valid commands and noise through voice activity detection, binds environmental visual features, vehicle motion parameters and positioning information to each command, and constructs command-scene pairing data. Based on acoustic models and pre-trained language models in the driving domain, it parses the core intent of commands, disambiguates ambiguous commands, and outputs structured commands.

4. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, The analysis of structured instructions clarifies the corresponding vehicle control module and key parameters; a three-dimensional verification system of safety, feasibility, and scenario adaptation is constructed based on multi-source real-time environmental data to determine whether the execution conditions are met; if met, the structured instructions are converted into parameterized execution signals; if not met, root cause analysis is used to locate the factors that do not meet the conditions and generate corresponding execution actions.

5. The intelligent driving cooperative control terminal based on a visual language action model according to claim 4, characterized in that, A three-dimensional verification system based on multi-source real-time environmental data is constructed, which includes safety constraint verification, such as verification of collision time (TTC), minimum safe distance, and traffic rule compliance. Feasibility verification includes verifying the vehicle's power, braking, and steering system capabilities as well as its adaptability to road conditions. Scene adaptation verification includes verifying the matching of instructions with the current road type and traffic flow status.

6. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, The process involves classifying and processing conflicting actions, identifying consecutive conflicting actions as conflicting actions, identifying the conflict type of each action, and classifying them into non-urgent and urgent actions. For non-urgent actions, execution is paused and resumed after the conflict is resolved. For urgent actions, parameters are adjusted to avoid conflict or replace conflicting actions with equivalent safe actions, and conflict coordination information is generated.

7. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, If there is no conflict, the time window of consecutive actions is analyzed, and the dependency relationship is identified by using the time alignment algorithm. Rigid constraints and flexible windows are marked, and the redundancy waiting of flexible windows is compressed based on the real-time environment. At the same time, the occupation of hardware resources such as computing power, sensors, and actuators by actions is analyzed, resource requirements are marked according to demand, and on-demand allocation and priority preemption mechanisms are adopted to generate normal collaborative processing information.

8. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, The environmental perception collaborative analysis method is as follows: the scene is classified, the collaborative strategy is automatically switched for different scenes, the real-time scene is obtained, and the corresponding scene classification result is determined. Then, the triggering condition is set based on the scene classification result, and the triggering condition includes the scene confidence. When the classification confidence of the target scene is greater than or equal to the switching threshold, the strategy conversion is initiated. For composite scenes, the corresponding composite strategy is triggered only when the confidence of all scenes meets the standard.

9. The intelligent driving cooperative control terminal based on a visual language action model according to claim 1, characterized in that, The generated environment-aware collaborative processing information includes: The system monitors risk indicators based on the current scenario classification results. These indicators include following safety distance, lateral intrusion risk, and intersection conflict risk. The system compares these indicators with risk thresholds, the specific values ​​of which are set by the operator. When a risk indicator exceeds the risk threshold, the system forcibly switches to a safety fallback strategy, which has higher priority than the regular scenario strategy. Simultaneously, the system monitors the risk indicator in real time. After three consecutive frames, the system returns the risk indicator to the safe range and gradually recovers from emergency braking to regular control. Once the transition is complete, the system automatically switches back to the original scenario strategy and generates environmental perception collaborative processing information.

10. The intelligent driving cooperative control terminal based on a visual language action model according to claim 9, characterized in that, The safety fallback strategy includes: Vehicle control: Trigger ABS+ESP coordinated braking to reduce the vehicle speed to a safe value or bring it to a stop, and simultaneously make a small lateral turn when there is space. Full-area warning: Activate hazard lights, honk the horn, flash the high-mounted brake light, and provide external voice announcements; System Lockdown: Freeze non-safe functions, allocate all computing power and executor permissions to risk avoidance control, and smoothly restore to the original strategy after the risk is eliminated.

Images