An intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling

Abstract

The present invention proposes an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling, which belongs to the field of autonomous driving technology. The system includes: a situational perception module, a feature extraction and fusion module, a semantic and action decision module, an adaptive cluster pick-up module, a trajectory generation and optimization module, and an embodied intelligence learning module. The autonomous driving technology of this system involves embodied intelligence, VLA multimodal architecture, etc., and on this basis, an end-to-end data processing method is implemented, so that the autonomous driving car can quickly generate corresponding strategies based on the input and respond according to the strategy. The autonomous driving vehicle that realizes environmental understanding, intelligent decision-making and precise control in complex scenarios is safer to drive.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling. Background Art

[0002] As a key application of artificial intelligence, autonomous driving technology is gradually becoming part of our lives and a key development direction for future transportation. It plays a vital role in improving traffic safety, optimizing traffic efficiency, and enriching travel options. Currently, autonomous driving technology has been widely applied in various fields, including public transportation, taxis, logistics and delivery, and urban infrastructure.

[0003] The basic process of autonomous driving consists of three parts: perception, decision-making, and control. The perception system integrates data from various sensors, and using various algorithms and supporting software, it determines the driving plan based on the perception layer's output. Ultimately, the control system controls the vehicle. Currently, there are two mainstream technical approaches: a multi-sensor fusion solution based on cameras, and a LiDAR-based solution with other sensors as a supplement.

[0004] Current autonomous vehicle systems typically focus on single-sensor and dynamic environment scheduling and transportation tasks, with little consideration given to the coordination between multiple sensing and dynamic environments. In high-density traffic or large-scale activity areas, challenges such as route conflicts, inefficiencies, and traffic congestion arise. Existing systems lack the multimodal perception and dynamic environment scheduling capabilities to cope with these complex scenarios. Consequently, autonomous vehicles lack the environmental understanding, intelligent decision-making, and precise control required to make driving and scheduling decisions in complex scenarios. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned problems existing in the existing technology and greatly improve its technical effect on the basis of the existing technology. The present invention provides an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling. The system includes: a situational awareness module, a feature extraction and fusion module, a semantic and action decision module, an adaptive cluster pick-up module, a trajectory generation and optimization module, and an embodied intelligent learning module.

[0006] The contextual awareness module is used to perceive the internal and external contextual data of the autonomous driving vehicle in real time, wherein the internal and external contextual data include passenger needs, traffic conditions, road conditions, weather data and vehicle status;

[0007] The feature extraction and fusion module is used to extract features from the internal and external context data, fuse the extracted features, and generate environmental features, wherein the environmental features at least include BEV space features;

[0008] The semantic and action decision module is used to generate intelligent behavior decisions by combining the environmental features in the feature extraction and fusion module through the VLA large model;

[0009] The adaptive cluster pick-up module is used to convert the final trajectory and intelligent behavior decisions into vehicle scheduling;

[0010] The trajectory generation and optimization module is used to build an Occupancy Network model of the environment occupancy grid in real time. The Occupancy Network predicts and updates the idle and occupied areas in the environment by combining the BEV spatial characteristics in the environmental characteristics. The Occupancy Network uses a self-supervised learning method to optimize the path planning and vehicle scheduling planning based on the idle and occupied areas. At the same time, reinforcement learning is introduced to optimize the real-time vehicle scheduling generated by the adaptive cluster pick-up module, and the optimization results are input into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0011] The embodied intelligent learning module is used to achieve real-time adjustment of the path planning and vehicle scheduling planning of the autonomous driving vehicle through the interaction between the autonomous driving vehicle and the environment, and to learn based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module, and input the learning results into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0012] Furthermore, the situational awareness module includes a passenger demand awareness submodule, a traffic dynamics awareness submodule, and a vehicle status awareness submodule;

[0013] The passenger demand perception submodule is used to obtain passengers' pick-up and drop-off requests, preferences, and estimated arrival times in real time, and generate passenger demand perception data based on passengers' historical behavior data;

[0014] The traffic dynamics perception submodule is used to obtain traffic flow in real time through traffic sensors, GPS maps and road cameras to generate traffic dynamics perception data; the traffic flow is used to analyze congestion or temporary road closures;

[0015] The vehicle status perception submodule is used to collect the power data, passenger number data and current pick-up and drop-off task progress data of each vehicle to generate vehicle status perception data.

[0016] Furthermore, the feature extraction and fusion module includes a feature extraction submodule and a fusion submodule;

[0017] The feature extraction submodule is used to extract features from the passenger demand perception data, traffic dynamics perception data and vehicle state perception data respectively to generate passenger demand perception feature data, traffic dynamics perception feature data and vehicle state perception feature data;

[0018] The fusion submodule is used to fuse passenger demand perception feature data, traffic dynamics perception feature data and vehicle status perception feature data using a multimodal Transformer model.

[0019] Furthermore, the specific implementation process of the fusion submodule includes:

[0020] Using a multimodal Transformer model, the corresponding feature data of passenger demand perception feature data, traffic dynamic perception feature data, and vehicle status perception feature data are fused through the self-attention mechanism in the multimodal Transformer model. The image features contained in the passenger demand perception feature data, traffic dynamic perception feature data, and vehicle status perception feature data are all extracted based on the bird's-eye view (BEV) combined with the Transformer model to generate BEV spatial features;

[0021] The self-attention mechanism formula of the multimodal Transformer model is:

[0022]

[0023] Among them, Q, K, V represent the query matrix, key matrix and value matrix respectively, d k represents the dimension of the key vector;

[0024] The formula for image feature extraction based on generating a bird's-eye view BEV combined with the Transformer model is:

[0025] P bev (X t )=Transformer bev (X t ),

[0026] Among them, X t represents the image sequence features in the passenger demand perception feature data, traffic dynamic perception feature data, and vehicle state perception feature data, P bev (X t ) represents a bird's-eye view of the output.

[0027] Furthermore, the semantic and action decision module includes: combining the environmental features in the feature extraction and fusion module through the VLA large model to obtain intelligent behavior decisions including target task scheduling, control instructions and target task scheduling execution strategies; the steps are: first, performing data fusion analysis and processing on the environmental features through the VLA large model to analyze the passenger needs and riding environment; second, combining the current road condition information and vehicle status information obtained by the traffic dynamic perception submodule and the vehicle status perception submodule to preliminarily generate the target task scheduling, and generate the corresponding control instructions and target task scheduling execution strategies for the vehicle based on the generated target task scheduling.

[0028] Furthermore, the embodied intelligent learning module includes: real-time perception and decision-making, environmental feedback mechanism, and reinforcement learning and self-supervised learning; the real-time perception and decision-making includes: obtaining real-time environmental information about the autonomous vehicle through the perception system, and making timely decisions based on the obtained information; the perception system integrates multiple sensors and obtains environmental data through the sensors, and establishes an environmental model by fusing and processing the sensor data. The established model formula is:

[0029] H t =F p (S t , v p )

[0030] Among them, H t is the environmental state at time t, and the environmental state is the internal and external situation data; F p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, v p are the parameters of the perception model;

[0031] The environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the scheduling task plan based on the feedback, thereby adjusting the scheduling task. First, the system generates corresponding decisions through the dynamic environment model established by the context perception module, and interacts with the target in the corresponding dynamic environment. Then, after interacting with the target in the dynamic environment, the system adjusts the scheduling task plan based on the feedback results and determines the next vehicle decision. Finally, the scheduling task plan is adjusted based on the readjusted scheduling task plan, and the final result is input into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0032] The reinforcement learning and self-supervised learning include: optimizing scheduling tasks through a reinforcement learning framework; optimizing the scheduling action value at the current moment by the scheduling action value at the previous moment; and continuously updating and learning based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module, and then selecting the best one, and inputting the final result into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0033] Furthermore, the trajectory generation and optimization module includes: Occ model construction and prediction and self-supervised learning;

[0034] The Occupancy model construction and prediction includes: based on the Occupancy Network, building an Occupancy model of the environment occupancy grid in real time. The Occupancy model predicts and updates the idle and occupied areas in the environment by combining the BEV spatial features in the environmental features. The training formula of the Occupancy Network is:

[0035] P occ (s t )=OccupancyNetwork(P bev (X t )), where P occ (s t ) represents the state of the occupied grid at time t, P bev (X t ) represents the bird’s-eye view output of the occupancy grid at time t;

[0036] The loss function of the OccupancyNetwork network is:

[0037] η(Occ t )=||Occ t -Occ' t || 2 ,

[0038] Among them Occ t is the environmental state at the current moment, Occ' t is the environmental state predicted by the system, || || is the norm, and η() is the function that minimizes the loss; the Occupancy Network adaptively learns from the environment and optimizes path planning in this way;

[0039] The self-supervised learning includes using a self-supervised learning method to optimize path planning and vehicle scheduling planning through idle and occupied areas, while introducing reinforcement learning to optimize vehicle scheduling, and inputting the optimization results into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0040] Furthermore, the semantic and action decision module, the adaptive cluster pick-up module, the trajectory generation and optimization module, and the embodied intelligent learning module are designed as an end-to-end automatic decision module, so that after the car surrounding data and passenger information data collected by the sensor are input into the automatic decision module, the automatic decision module directly generates a series of driving decisions, which include: task planning, path planning, control instructions, task scheduling, and scheduling optimization.

[0041] The beneficial effects of the present invention are:

[0042] This invention provides an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling, which has the following advantages:

[0043] 1. The present invention uses the self-attention mechanism of the multimodal Transformer model to extract features from internal and external context data; compared with single processing of internal and external context data, the internal and external context data obtained by this method will be more accurate.

[0044] 2. This invention provides an autonomous driving pick-up and dispatch system that combines embodied intelligence, a VLA multimodal architecture, and end-to-end optimization from environmental perception to task execution. Its adaptive and dispatching capabilities will be greatly enhanced, making autonomous driving vehicles safer.

[0045] 3. Self-supervised learning and occupancy network are used to optimize the path optimization planning and vehicle scheduling optimization planning of autonomous driving vehicles, thereby improving the safety of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to the present invention. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention and cannot be used to limit the present invention.

[0048] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0049] like Figure 1FIG2 is a schematic diagram of an intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to an embodiment of the present invention; the schematic diagram includes: a context perception module A1, a feature extraction and fusion module A2, a semantic and action decision module A3, an adaptive cluster pick-up module A4, a trajectory generation and optimization module A5, and an embodied intelligent learning module A6;

[0050] The context perception module A1 is used to perceive the internal and external context data of the autonomous driving vehicle in real time, wherein the internal and external context data include passenger needs, traffic conditions, road conditions, weather data and vehicle status;

[0051] The feature extraction and fusion module A2 is used to extract features from the internal and external context data, fuse the extracted features, and generate environmental features, wherein the environmental features at least include BEV space features;

[0052] The semantic and action decision module A3 is used to generate intelligent behavior decisions by combining the environmental features in the feature extraction and fusion module A2 through the VLA large model;

[0053] The adaptive cluster pick-up module A4 is used to convert the final trajectory and intelligent behavior decision into vehicle scheduling;

[0054] The trajectory generation and optimization module A5 is used to build an Occupancy Network model of the environment occupancy grid in real time. The Occupancy Network predicts and updates the idle and occupied areas in the environment by combining the BEV spatial characteristics in the environmental characteristics. The Occupancy Network uses a self-supervised learning method to optimize the path planning and vehicle scheduling planning based on the idle and occupied areas. At the same time, reinforcement learning is introduced to optimize the real-time vehicle scheduling generated by the adaptive cluster pick-up module, and the optimization results are input into the adaptive cluster pick-up module A4 in real time for real-time vehicle scheduling.

[0055] The embodied intelligent learning module A6 is used to achieve real-time adjustment of the path planning and vehicle scheduling planning of the autonomous driving vehicle through the interaction between the autonomous driving vehicle and the environment, and to learn based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module A5, and input the learning results into the adaptive cluster pick-up module A4 in real time for real-time vehicle scheduling.

[0056] In the above embodiment, specifically, the situational awareness module A1 includes a passenger demand awareness submodule, a traffic dynamics awareness submodule, and a vehicle status awareness submodule;

[0057] The passenger demand perception submodule is used to obtain passengers' pick-up and drop-off requests, preferences, and estimated arrival times in real time, and generate passenger demand perception data based on passengers' historical behavior data;

[0058] The traffic dynamics perception submodule is used to obtain traffic flow in real time through traffic sensors, GPS maps and road cameras to generate traffic dynamics perception data; the traffic flow is used to analyze congestion or temporary road closures;

[0059] The vehicle status perception submodule is used to collect the power data, passenger number data and current pick-up and drop-off task progress data of each vehicle to generate vehicle status perception data.

[0060] In the above embodiment, specifically, the feature extraction and fusion module A2 includes a feature extraction submodule and a fusion submodule;

[0061] The feature extraction submodule is used to extract features from the passenger demand perception data, traffic dynamics perception data and vehicle state perception data respectively to generate passenger demand perception feature data, traffic dynamics perception feature data and vehicle state perception feature data;

[0062] The fusion submodule is used to fuse the passenger demand perception feature data, the traffic dynamics perception feature data and the vehicle state perception feature data using a multimodal Transformer model;

[0063] Specifically, the feature extraction process of passenger demand perception data is as follows:

[0064] Obtaining passenger pick-up requests, passenger preferences, and passenger historical behavior data through the passenger's mobile device, wherein the passenger historical behavior data includes at least the passenger's historical travel patterns, and the passenger historical behavior data is stored in an InfluxDB database;

[0065] The temporal convolutional network (TCN) model is used to extract features from passenger pick-up requests, passenger preferences, and passenger historical behavior data.

[0066] Specifically, the feature extraction process of traffic dynamic perception data is as follows:

[0067] The traffic dynamics perception submodule includes a model building unit and a model analysis unit;

[0068] The specific implementation process of the model building unit is as follows:

[0069] The traffic flow is modeled using a graph neural network (GNN) to generate a graph neural network GNN model. The graph neural network GNN model represents the traffic network as a graph structure, where nodes represent road sections and edges represent traffic flow. The graph neural network GNN model propagates information through graph convolution operations. The calculation formula of the graph neural network GNN model is:

[0070]

[0071] in, represents the hidden state of node i in the mth layer, N(i) represents the neighbor set of node i, and W m It represents the weight matrix of the mth layer, δ represents the activation function, d i d j denote the degrees of nodes i and j respectively;

[0072] Specifically, the feature extraction implementation process of the graph neural network GNN model is as follows:

[0073] Combined with the vehicle status and road condition information reported by the vehicles, the graph neural network (GNN) model uses the average speed and load of the road section to call the K-Means++ mean clustering algorithm to analyze congestion or temporary road closures, and generate traffic dynamic perception feature data;

[0074] The specific implementation process of feature extraction of vehicle state perception data is as follows:

[0075] The edge computing unit embedded in each vehicle is responsible for local calculation and regular uploading of vehicle internal status data, including onboard CAN bus data, battery level, and current task progress;

[0076] The dynamic Bayesian network (DBN) model is used to extract features from the vehicle's internal state data and generate vehicle state perception feature data.

[0077] In the above embodiment, specifically, the specific implementation process of the fusion submodule includes:

[0078] A multimodal Transformer model is used to fuse passenger demand perception feature data, traffic dynamics perception feature data, and vehicle status perception feature data through the self-attention mechanism in the multimodal Transformer model. The image features contained in the passenger demand perception feature data, traffic dynamics perception feature data, and vehicle status perception feature data are extracted based on the bird's-eye view (BEV) combined with the Transformer model to generate BEV spatial features.

[0079] The self-attention mechanism formula of the multimodal Transformer model is:

[0080]

[0081] Among them, Q, K, V represent the query matrix, key matrix and value matrix respectively, d k represents the dimension of the key vector;

[0082] The formula for image feature extraction based on generating a bird's-eye view BEV combined with the Transformer model is:

[0083] P bev (X t )=Transformer bev (X t ),

[0084] Among them, X t represents the image sequence features in the passenger demand perception feature data, traffic dynamic perception feature data, and vehicle state perception feature data, P bev (X t ) represents a bird's-eye view of the output.

[0085] In the above embodiment, specifically, the semantic and action decision module A3 includes: combining the environmental features in the feature extraction and fusion module through the VLA large model to obtain intelligent behavior decisions including target task scheduling, control instructions and target task scheduling execution strategies; the steps are: first, performing data fusion analysis and processing on the environmental features through the VLA large model to analyze the needs of passengers and the riding environment; secondly, combining the current road condition information and the vehicle's own status information obtained by the traffic dynamic perception submodule and the vehicle status perception submodule to preliminarily generate the target task scheduling, and generate the corresponding control instructions and target task scheduling execution strategies for the vehicle based on the generated target task scheduling.

[0086] In the above embodiment, specifically, the embodied intelligent learning module A6 includes: real-time perception and decision-making, environmental feedback mechanism, and reinforcement learning and self-supervised learning; the real-time perception and decision-making includes: obtaining real-time environmental information about the autonomous vehicle through the perception system, and making timely decisions based on the obtained information; the perception system integrates multiple sensors and obtains environmental data through the sensors, and establishes an environmental model by fusing and processing the sensor data. The established model formula is:

[0087] H t =F p (S t , v p )

[0088] Among them, H t is the environmental state at time t, and the environmental state is the internal and external situation data; F p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, v p are the parameters of the perception model;

[0089] The environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the scheduling task plan based on the feedback, thereby adjusting the scheduling task. First, the system generates corresponding decisions through the dynamic environment model established by the context perception module A1, and interacts with the target in the corresponding dynamic environment. Then, after interacting with the target in the dynamic environment, the system adjusts the scheduling task plan based on the feedback results and determines the next vehicle decision. Finally, the scheduling task plan is adjusted based on the readjusted scheduling task plan, and the final result is input into the adaptive cluster pick-up module A4 in real time for real-time vehicle scheduling.

[0090] The reinforcement learning and self-supervised learning include: optimizing scheduling tasks through a reinforcement learning framework; optimizing the scheduling action value at the current moment by the scheduling action value at the previous moment; and continuously updating and learning based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module A5, and then selecting the best one, and inputting the final result into the adaptive cluster pick-up module A4 in real time for real-time vehicle scheduling.

[0091] In the above embodiment, specifically, the trajectory generation and optimization module A5 includes: Occ model construction and prediction and self-supervised learning;

[0092] The Occupancy model construction and prediction includes: based on the Occupancy Network, building an Occupancy model of the environmental occupancy grid in real time. The Occupancy model predicts and updates the idle and occupied areas in the environment by combining the BEV spatial features in the environmental features. The training formula of the Occupancy Network is:

[0093] P occ (s t )=OccupancyNetwork(P bev (X t )),

[0094] Among them, P occ (s t ) represents the state of the occupied grid at time (t), P bev (X t ) represents the bird’s-eye view output of the occupancy grid at time (t);

[0095] The loss function of the Occupancy Network is:

[0096] η(Occ t )=||Occ t -Occ' t || 2 ,

[0097] Among them Occ t is the environmental state at the current moment, Occ' t is the environmental state predicted by the system, || || is the norm, and η() is the function that minimizes the loss; the Occupancy Network adaptively learns from the environment and optimizes path planning in this way;

[0098] The self-supervised learning includes using a self-supervised learning method to optimize path planning and vehicle scheduling planning through idle and occupied areas, while introducing reinforcement learning to optimize vehicle scheduling, and inputting the optimization results into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

[0099] In the above embodiment, specifically, the semantic and action decision module A3, the adaptive cluster pick-up module A4, the trajectory generation and optimization module A5 and the embodied intelligent learning module A6 are designed as an end-to-end automatic decision module, so that after the car surrounding data and passenger information data collected by the sensor are input into the automatic decision module, the automatic decision module directly generates a series of driving decisions, and the series of driving decisions include: task planning, path planning, control instructions, task scheduling and scheduling optimization.

[0100] It should be noted that the above-mentioned end-to-end automatic decision-making module is built through OpenPilot open source autonomous driving technology. Through sensor data collection and module output, decisions for controlling autonomous driving vehicles are directly generated, making decisions faster and more accurate.

[0101] In the above embodiment, specifically, the adaptive cluster pick-up module converts the final trajectory and behavior strategy into a vehicle scheduling plan, and the mathematical calculation formula is:

[0102]

[0103] Constraints:

[0104]

[0105] Among them, w ij represents the task allocation matrix, d ij represents the distance between the vehicle and the task, min() represents the final trajectory and behavior strategy optimization function, i and j represent nodes i and j respectively, N and M represent the total number of nodes i and j respectively;

[0106] If multiple optimal trajectories are selected, the similarity between different driving paths is compared by using a trajectory similarity measurement method to select the optimal driving path. The trajectory similarity measurement methods implemented in this embodiment include Euclidean distance, dynamic time warping, etc.

Claims

1. An intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling, characterized by: The system includes a context perception module, a feature extraction and fusion module, a semantic and action decision module, an adaptive cluster pick-up module, a trajectory generation and optimization module, and an embodied intelligent learning module; The contextual awareness module is used to perceive the internal and external contextual data of the autonomous driving vehicle in real time, wherein the internal and external contextual data include passenger needs, traffic conditions, road conditions, weather data and vehicle status; The feature extraction and fusion module is used to extract features from the internal and external context data, fuse the extracted features, and generate environmental features, wherein the environmental features at least include BEV space features; The semantic and action decision module is used to generate intelligent behavior decisions by combining the environmental features in the feature extraction and fusion module through the VLA large model; The adaptive cluster pick-up module is used to convert the final trajectory and intelligent behavior decisions into vehicle scheduling; The trajectory generation and optimization module is used to build an Occupancy Network model of the environment occupancy grid in real time. The Occupancy Network predicts and updates the idle and occupied areas in the environment by combining the BEV spatial characteristics in the environmental characteristics. The Occupancy Network uses a self-supervised learning method to optimize the path planning and vehicle scheduling planning based on the idle and occupied areas. At the same time, reinforcement learning is introduced to optimize the real-time vehicle scheduling generated by the adaptive cluster pick-up module, and the optimization results are input into the adaptive cluster pick-up module in real time for real-time vehicle scheduling. The embodied intelligent learning module is used to adjust the path planning and vehicle scheduling planning of the autonomous driving vehicle in real time through the interaction between the autonomous driving vehicle and the environment, and learn based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module, and input the learning results into the adaptive cluster pick-up module in real time for real-time vehicle scheduling; The embodied intelligent learning module includes: real-time perception and decision-making, environmental feedback mechanism, reinforcement learning and self-supervised learning. The real-time perception and decision-making includes: obtaining real-time information about the autonomous vehicle's surrounding environment through the perception system, and making timely decisions based on the obtained information. The perception system integrates multiple sensors and obtains surrounding environment data through sensors. By fusing and processing sensor data, an environmental model is established. The established model formula is: H t =F p (S t ,v p ) Among them, H t is the environmental state at time t, and the environmental state is the internal and external situation data; F p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, v p are the parameters of the perception model; The environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the scheduling task plan based on the feedback, thereby adjusting the scheduling task. First, the system generates corresponding decisions through the dynamic environment model established by the context perception module, and interacts with the target in the corresponding dynamic environment. Then, after interacting with the target in the dynamic environment, the system adjusts the scheduling task plan based on the feedback results and determines the next vehicle decision. Finally, the scheduling task plan is adjusted based on the readjusted scheduling task plan, and the final result is input into the adaptive cluster pick-up module in real time for real-time vehicle scheduling. The reinforcement learning and self-supervised learning include: optimizing scheduling tasks through a reinforcement learning framework; optimizing the scheduling action value at the current moment by the scheduling action value at the previous moment; and continuously updating and learning based on the received real-time environmental feedback, the environmental characteristics, and the optimization results of the trajectory generation and optimization module, and then selecting the best one, and inputting the final result into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

2. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 1 is characterized in that: The situational awareness module includes a passenger demand awareness submodule, a traffic dynamics awareness submodule, and a vehicle status awareness submodule; The passenger demand perception submodule is used to obtain passengers' pick-up and drop-off requests, preferences, and estimated arrival times in real time, and generate passenger demand perception data based on passengers' historical behavior data; The traffic dynamics perception submodule is used to obtain traffic flow in real time through traffic sensors, GPS maps and road cameras to generate traffic dynamics perception data; the traffic flow is used to analyze congestion or temporary road closures; The vehicle status perception submodule is used to collect the power data, passenger number data and current pick-up and drop-off task progress data of each vehicle to generate vehicle status perception data.

3. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 2 is characterized in that: The feature extraction and fusion module includes a feature extraction submodule and a fusion submodule; The feature extraction submodule is used to extract features from the passenger demand perception data, traffic dynamics perception data and vehicle state perception data respectively to generate passenger demand perception feature data, traffic dynamics perception feature data and vehicle state perception feature data; The fusion submodule is used to fuse passenger demand perception feature data, traffic dynamics perception feature data and vehicle status perception feature data using a multimodal Transformer model.

4. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 3 is characterized in that: The specific implementation process of the fusion submodule includes: A multimodal Transformer model is used to fuse passenger demand perception feature data, traffic dynamics perception feature data, and vehicle status perception feature data through the self-attention mechanism in the multimodal Transformer model. The image features contained in the passenger demand perception feature data, traffic dynamics perception feature data, and vehicle status perception feature data are extracted based on the bird's-eye view (BEV) combined with the Transformer model to generate BEV spatial features. The self-attention mechanism formula of the multimodal Transformer model is: Among them, Q, K, V represent the query matrix, key matrix and value matrix respectively, d k represents the dimension of the key vector; The formula for image feature extraction based on generating a bird's-eye view BEV combined with the Transformer model is: P bev (X t )=Transformer bev (X t ), Among them, X t represents the image sequence features in the passenger demand perception feature data, traffic dynamic perception feature data, and vehicle state perception feature data, P bev (X t ) represents a bird's-eye view of the output.

5. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 4 is characterized in that: The semantic and action decision module includes: combining the environmental features in the feature extraction and fusion module with the VLA large model to obtain intelligent behavior decisions including target task scheduling, control instructions and target task scheduling execution strategy; the steps are: first, using the VLA large model to perform data fusion analysis and processing on the environmental features to analyze passenger needs and the riding environment; second, combining the current road condition information and vehicle status information obtained by the traffic dynamic perception submodule and the vehicle status perception submodule to preliminarily generate the target task scheduling, and generating the corresponding control instructions and target task scheduling execution strategy for the vehicle based on the generated target task scheduling.

6. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 1 is characterized in that: The trajectory generation and optimization module includes: Occ model construction and prediction and self-supervised learning; The Occupancy model construction and prediction includes: based on the Occupancy Network, building an Occupancy model of the environmental occupancy grid in real time. The Occupancy model predicts and updates the idle and occupied areas in the environment by combining the BEV spatial features in the environmental features. The training formula of the Occupancy Network is: P occ (s t )=OccupancyNetwork(P bev (X t )), Among them, P occ (s t ) represents the state of the occupied grid at time t, P bev (X t ) represents the bird’s-eye view output of the occupancy grid at time t; The loss function of the OccupancyNetwork network is: η(Occ t )=||Occ t -Occ' t || 2 , Among them Occ t is the environmental state at the current moment, Occ' t is the environmental state predicted by the system, |||| is the norm, and η() is the function that minimizes the loss; the Occupancy Network adaptively learns from the environment and optimizes path planning in this way; The self-supervised learning includes using a self-supervised learning method to optimize path planning and vehicle scheduling planning through idle and occupied areas, while introducing reinforcement learning to optimize the real-time vehicle scheduling generated by the adaptive cluster pick-up module, and inputting the optimization results into the adaptive cluster pick-up module in real time for real-time vehicle scheduling.

7. The intelligent autonomous driving vehicle dispatching system based on multimodal perception and dynamic environment modeling according to claim 1 is characterized in that: The semantic and action decision module, the adaptive cluster pick-up module, the trajectory generation and optimization module, and the embodied intelligent learning module are designed as an end-to-end automatic decision module. After the car surrounding data and passenger information data collected by sensors are input into the automatic decision module, the automatic decision module directly generates a series of driving decisions. The series of driving decisions includes: task planning, path planning, control instructions, task scheduling, and scheduling optimization.

Images