Automatic driving vehicle dynamic pick-up management system based on multi-modal large model

Abstract

The invention provides an automatic driving vehicle dynamic pick-up management system based on a multi-modal large model, and belongs to the technical field of automatic driving. A multi-modal large model processing module; a personalized pick-up point recommendation module; a dynamic path planning and pick-up point adjusting module; an intelligent passenger interaction and feedback module; the system aims at providing intelligent pick-up and drop-off point selection and dynamic adjustment capability for an automatic driving vehicle. According to the system, efficient perception and real-time response to a complex environment are realized by fusing multi-modal data, and an optimal path and a pick-up point are generated in real time; the system can automatically adjust pick-up points and path planning according to real-time traffic, environment changes and passenger demands, and provides personalized passenger experience.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a dynamic pick-up and drop-off management system for autonomous driving vehicles based on a multimodal large model. Background Art

[0002] With the rapid development of autonomous driving technology, automatic passenger pick-up services have gradually been proposed; the automatic passenger pick-up services achieved through autonomous driving technology have the advantages of improving traffic safety, improving traffic efficiency, improving passenger experience, saving time and promoting the intelligence of the transportation system.

[0003] However, there are various problems in the operation of autonomous vehicles. For example, the route selected by the analysis system of existing autonomous vehicles is not the best route and cannot be adjusted in real time; the pick-up point location selected by the autonomous vehicle is not what the passengers need; the pick-up point selected by the autonomous vehicle cannot be adjusted in real time according to passenger needs and environmental changes.

[0004] Therefore, it is necessary to improve the automatic pick-up and drop-off system of autonomous vehicles so that the improved system can not only generate accurate pick-up routes and pick-up points, but also adjust and update the pick-up routes and pick-up points in real time according to environmental changes and passenger needs. Summary of the invention

[0005] The purpose of the present invention is to overcome the above problems existing in the prior art and to greatly improve the technical effect thereof on the basis of the original technology; for this purpose, the present invention provides a dynamic pick-up and drop-off management system for autonomous driving vehicles based on a multimodal large model, the system comprising:

[0006] Multimodal data acquisition module; multimodal large model processing module; personalized pick-up point recommendation module; dynamic route planning and pick-up point adjustment module; intelligent passenger interaction and feedback module.

[0007] The multimodal data acquisition module is used to receive the passenger's voice, text, image and video input information, obtain the real-time environmental data collected by the vehicle sensor, and perform feature extraction on the acquired data; then, the original data and all the data after feature extraction are passed to the multimodal large model processing module; specifically, the multimodal data acquisition module includes: data acquisition and data feature extraction; data acquisition refers to obtaining passenger information and real-time environmental data through various sensors of the self-driving car; data feature extraction refers to obtaining feature data of the corresponding data by extracting features from the collected data; data feature extraction is divided into: voice input processing, image and video input processing and sensor data processing; voice input processing adopts an end-to-end ASR model; the end-to-end ASR model is a speech recognition method that attempts to output text directly from the original voice signal through a unified neural network model, omitting the traditional AS The ASR model consists of multiple independent steps; that is, the end-to-end ASR model integrates the processing of speech information including the acoustic model, speech model and decoder; the steps of the end-to-end ASR model to process speech information are as follows: pre-training stage: through unsupervised training, the model learns the characteristics of speech signals to generate potential representations, thereby capturing the speech patterns in the audio; fine-tuning stage: using labeled data to further optimize the model so that the model can accurately map from speech signals to text; decoding stage: extracting the final speech-to-text conversion process from the trained model through the Transformer decoder in the model, generating a series of possible text outputs based on the input speech features and the conversion process, and obtaining the final recognition result by selecting the most likely sequence; the speech formula for converting speech signals to text is:

[0008]

[0009] Where X represents the speech feature sequence, X = (x1, x2, ..., x T ), T means there are T time steps in X; W means the predicted word sequence, P(W|X) means the conditional probability of predicting the word sequence W given the speech sequence X; x t represents the speech feature at the current time step t, ω t represents the target output at the current time step t, h t－1 represents the hidden state of the previous time step t-1, θ refers to the model parameters, P(ω t x t ,h t－1 ,θ) indicates the prediction ω at the current time step t t The conditional probability of .

[0010] The image and video input processing adopts the feature extraction based on the convolutional neural network CNN and the visual transformer ViT, and adopts the YOLOv5 model to identify the target after the feature extraction; the steps of the image and video input processing are: the image and video information is preliminarily processed by the convolutional neural network CNN to extract the low-level features of the image and video information, and the low-level features are the local features of the image and video information; the low-level features extracted by the convolutional neural network CNN are transferred to the visual transformer ViT module, and the visual transformer ViT module further captures the global information by using the self-attention mechanism to identify the long-distance dependency and context information; combining the features extracted by the convolutional neural network CNN and the visual transformer ViT module, the YOLOv5 model divides the input image into multiple grids according to the combined features, each grid is responsible for predicting the object in the image, and giving the category and bounding box; target recognition is performed by combining the first-order residual structure with multi-scale feature fusion; the first-order residual structure refers to directly adding the input and output of the feature extraction by introducing skip connections, so that the information can bypass some convolutional layers through these skip connections and be directly transferred to deeper layers; the multi-scale feature fusion adopts the feature pyramid network.

[0011] Sensor data processing combines PointNet++ processing and VoxelNet algorithm for processing; first, PointNet++ obtains point cloud data through hierarchical data sampling of the data in the sensor, then converts the point cloud data into voxel data through the VoxelNet algorithm, and processes the voxel data through the three-dimensional convolutional neural network 3D CNN technology to obtain low-level local spatial features; finally, PointNet++ aggregates the features extracted from the 3D CNN, learns the global features, and combines the local spatial features with the global features to generate the 3D structural features of the environment model.

[0012] The multimodal large model processing module is used to analyze and fuse the data collected by the multimodal data collection module to generate intelligent decisions for pick-up point recommendation and route planning; specifically, the multimodal large model processing module includes: natural language processing NLP, image and video processing, environmental perception and 3D space understanding and language emotion recognition; the natural language processing NLP understands and processes feature text information through the BERT large language model and the method of introducing multi-task learning MTL, and the processing steps are: BERT pre-training, BERT pre-training through the large model autonomous driving corpus, BERT learns a general language representation; fine-tuning BERT and multi-task learning MTL, adding multiple tasks including classification, proposition entity recognition and question answering to BERT, and optimizing BERT's shared layer and task-specific head; joint loss optimization, optimizing the goals of different characters through the joint loss function, and enhancing the generalization ability of the model; according to the needs of downstream tasks, the corresponding results are output through the layer corresponding to the task; the BER T pre-training model formula is:

[0013] H t = Transformer(H t－1 , H t－2 , ..., H0; θ)

[0014] Among them, H t represents the hidden state of the Transformer at the tth time step, H t－1 represents the hidden state of the Transformer at the t-1th time step, where the time step refers to the representation of each unit time in a discretized time series; θ represents the model parameters.

[0015] The image and video processing is based on the image and video processing of the multimodal data acquisition module, and adds a three-dimensional convolutional neural network 3D-CNN to capture the dynamic characteristics of the target; the steps are: the three-dimensional convolutional neural network 3D-CNN sets a one-dimensional feature extraction structure and a three-dimensional feature extraction structure; obtains the target feature through the one-dimensional feature extraction structure, and extracts the spatiotemporal features of the target that change with time through the three-dimensional feature extraction structure; the extracted target features and the corresponding spatiotemporal features are combined to capture the dynamic characteristics of the target.

[0016] The environmental perception and 3D space understanding use LiDAR point cloud processing combined with simultaneous positioning and map construction SLAM technology to generate a real-time 3D environmental map; the steps of generating a real-time 3D environmental map are: extracting local features of point cloud data through LiDAR point cloud processing in sequence through the steps of: acquiring point cloud data, preprocessing point cloud data and extracting features from the preprocessed point cloud data; continuously accumulating point cloud data features and updating positions at different times through map construction SLAM technology to construct a 3D environmental map, and comparing the point cloud obtained by each LiDAR scan with the data of the previous scan to gradually update the 3D environmental map.

[0017] The language emotion recognition refers to classifying speech emotions through an LSTM model based on emotion vectors, and adjusting the pick-up point priority according to the classification results; the emotion recognition model formula is:

[0018] s t =σ(W s ·x t +U s ·h t－1 +b s )

[0019] Among them, s t represents the emotional state, σ represents the Sigmoid activation function, W s and U s Represents the weight parameter of the model, x t represents the input speech feature vector, h t－1 is the hidden state at the previous moment, b s Represents the bias term, which is used to adjust the model output.

[0020] Furthermore, by processing various types of data, different types of data are fused and analyzed to lay the foundation for subsequent route planning and determination of pick-up and drop-off points.

[0021] The personalized pick-up point recommendation module is used to generate personalized pick-up points based on the passenger's historical behavior, preferences and current environment; specifically, a customized pick-up point service is provided based on the passenger's historical behavior through a collaborative filtering algorithm; the process is to use matrix decomposition to decompose the passenger and pick-up point score matrix into multiple low-dimensional matrices, so as to determine which pick-up points are more convenient for the passenger; the pick-up point location is determined by the score value obtained by the inner product of the characteristic vector of the passenger and the pick-up point, and the pick-up point with the highest score value of the passenger characteristic vector is used as the best pick-up point; after the pick-up point is determined, the pick-up point is adjusted according to environmental changes and passenger needs to ensure the best experience.

[0022] The dynamic path planning and pick-up point adjustment module is used to dynamically adjust the pick-up point and path planning in combination with real-time environmental data and multimodal input; dynamically adjust the path planning and pick-up point according to the dynamic changes of the real-time environment; the dynamic path planning and pick-up point adjustment module includes: real-time environmental data including traffic flow, road closures and weather; dynamic path planning refers to the system using advanced path planning algorithms to select the optimal driving route, while supporting dynamic adjustment of pick-up points to ensure that passengers can get on and off the vehicle quickly and safely; and when roads are closed, traffic is congested and passengers request the location of the pick-up point, the system can adjust the pick-up point location in real time and feed back the adjusted pick-up point location to the passengers; use the A* path planning algorithm and reinforcement learning technology to ensure that the vehicle can select the optimal path according to changes in the external environment; the A* path planning algorithm is a heuristic path search algorithm used to quickly plan the optimal path under a known map and environment; the system generates the shortest path based on the A* algorithm in combination with real-time traffic data; the A* path planning algorithm formula is:

[0023] f(n)=g(n)+h(n)

[0024] Among them, n represents the node, which represents a certain state and position of the automatic car, g(n) represents the cost from the starting point to the current node, h(n) is the estimated cost from the current node to the target point, and f(n) is the total cost.

[0025] After planning the optimal path through the A* path planning algorithm, the path needs to be optimized through reinforcement learning, and the Q-learning algorithm in deep reinforcement learning DRL is used to achieve real-time path optimization of the vehicle in a dynamic environment.

[0026] The Q-learning algorithm formula is:

[0027]

[0028] Where Q(s, a) is the state-action value function, which refers to the maximum expected cumulative reward that the autonomous car can obtain by taking action a in state s, r is the immediate reward, γ is a discount factor between 0 and 1, s' refers to the next state to which the autonomous car transfers after taking action a, and a' refers to the possible actions that the autonomous car may take in state s'. It refers to the maximum value of the state-action value function of all possible actions a' in the next state s'.

[0029] The intelligent passenger interaction and feedback module is used to interact with passengers in real time, share the driving route and recommended pick-up points with passengers, and passengers view the driving route and recommended pick-up points and propose new requirements based on the needs. The system feeds back the new requirements to the dynamic route planning and pick-up point adjustment module, and the dynamic route planning and pick-up point adjustment module adjusts the route and pick-up point according to the new requirements of the passengers; specifically, the intelligent passenger interaction and feedback module includes: the system interacts with passengers through their mobile devices to provide real-time pick-up point suggestions and route updates; passengers view the recommended pick-up points, the real-time location of the vehicle and the estimated arrival time through mobile applications; the system also uses augmented reality AR navigation to help passengers quickly find the vehicle parking location in a complex environment; the visual SLAM algorithm is combined with the camera data of the passenger's device to provide accurate AR navigation; the formula for AR navigation is:

[0030] T cw =K[R|t]

[0031] Among them, T cw is the transformation matrix from camera coordinates to world coordinates, and the world coordinates are used to describe the absolute position of objects in the scene; K is the camera intrinsic parameter matrix, R and t are the rotation matrix and translation vector respectively; by determining the absolute positions of objects, passengers and autonomous vehicles in the camera, the position and scene information are mapped to the AR scene, and passengers navigate through the AR rendered interface to find the pick-up point location of the autonomous vehicle.

[0032] The beneficial effects of the present invention are:

[0033] The present invention provides a dynamic pick-up and drop-off management system for autonomous driving vehicles based on a multimodal large model, the system comprising a multimodal data acquisition module; a multimodal large model processing module; a personalized pick-up and drop-off point recommendation module; a dynamic path planning and pick-up and drop-off point adjustment module; and an intelligent passenger interaction and feedback module. The system aims to provide autonomous driving vehicles with intelligent pick-up and drop-off point selection and dynamic adjustment capabilities. The system achieves efficient perception and real-time response to complex environments by fusing multimodal data, and generates optimal paths and pick-up and drop-off points in real time. The system can automatically adjust pick-up and drop-off points and path planning according to real-time traffic, environmental changes, and passenger needs, and provide a personalized passenger experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the dynamic pick-up and drop-off management system of the autonomous driving vehicle based on the multimodal large model of the present invention.

[0035] Figure 2 This is a flow chart of the dynamic pick-up and drop-off of an autonomous driving vehicle based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0036] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention, and cannot be used to limit the present invention.

[0037] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0038] like Figure 1 The present invention is a schematic diagram of a dynamic pick-up and drop-off management system for an autonomous driving vehicle based on a multimodal large model according to an embodiment of the present invention, which includes: a multimodal data acquisition module S100; a multimodal large model processing module S200; a personalized pick-up point recommendation module S300; a dynamic path planning and pick-up point adjustment module S400; and an intelligent passenger interaction and feedback module S500.

[0039] Specifically, the multimodal data acquisition module S100 is used to receive voice, text, image and video input information from passengers, obtain real-time environmental data collected by vehicle sensors, and perform feature extraction on the acquired data; subsequently, the original data and all data after feature extraction are passed to the multimodal large model processing module S200.

[0040] Furthermore, the multimodal data acquisition module S100 includes: data acquisition and data feature extraction; the data acquisition refers to obtaining passenger information and real-time environmental data through various sensors of the self-driving car; the data feature extraction refers to obtaining feature data of the corresponding data by extracting features from the collected data; the data feature extraction is divided into: voice input processing, image and video input processing and sensor data processing; the voice input processing adopts an end-to-end ASR model; the end-to-end ASR model is a speech recognition method that attempts to output text directly from the original voice signal through a unified neural network model, omitting multiple independent steps in the traditional ASR model; that is, the end-to-end ASR model will include an acoustic model, a voice The model and decoder agree on the processing of speech information; the steps of the end-to-end ASR model to process speech information are as follows: pre-training stage: through unsupervised training, the model learns the characteristics of speech signals to generate potential representations, thereby capturing the speech patterns in the audio; fine-tuning stage: using labeled data to further optimize the model so that the model can accurately map from speech signals to text; decoding stage: extracting the final speech-to-text conversion process from the trained model through the Transformer decoder in the model, generating a series of possible text outputs based on the input speech features and the conversion process, and obtaining the final recognition result by selecting the most likely sequence; the speech formula for converting speech signals to text is:

[0041]

[0042] Where X represents the speech feature sequence, X = (x1, x2, ..., x T ), T means there are T time steps in X; W means the predicted word sequence, P(W|X) means the conditional probability of predicting the word sequence W given the speech sequence X; x t represents the speech feature at the current time step t, ω t represents the target output at the current time step t, h t－1 represents the hidden state of the previous time step t-1, θ refers to the model parameters, P(ω t |x t ,h t－1 ,θ) indicates the prediction ω at the current time step t t The conditional probability of .

[0043] The image and video input processing adopts feature extraction based on convolutional neural network CNN and visual transformer ViT, and adopts YOLOv5 model to identify the target after feature extraction; the steps of image and video input processing are: preliminarily processing the image and video information through convolutional neural network CNN, extracting low-level features of the image and video information, and the low-level features are local features of the image and video information; the low-level features extracted by the convolutional neural network CNN are transferred to the visual transformer ViT module, and the visual transformer ViT module further captures global information by using the self-attention mechanism, and identifies long-distance dependencies and context information; combining the features extracted by the convolutional neural network CNN and the visual transformer ViT module, the YOLOv5 model divides the input image into multiple grids according to the combined features, each grid is responsible for predicting the object in the image, and giving the category and bounding box; target recognition is performed by combining the first-order residual structure with multi-scale feature fusion; the first-order residual structure refers to directly adding the input and output of feature extraction by introducing skip connections, so that information can bypass some convolution layers through these skip connections and be directly transferred to deeper layers; the multi-scale feature fusion adopts feature pyramid network.

[0044] The sensor data processing combines PointNet++ processing and VoxelNet algorithm for processing; first, point cloud data is obtained by hierarchical data sampling of data in the sensor through PointNet++, and then the point cloud data is converted into voxel Voxel data through the VoxelNet algorithm, and the voxel data is processed through the three-dimensional convolutional neural network 3D CNN technology to obtain low-level local spatial features; finally, PointNet++ aggregates the features extracted from 3DCNN, learns global features, and combines local spatial features with global features to generate 3D structural features of the environment model.

[0045] In the above embodiment, the multimodal large model processing module S200 is used to analyze and fuse the data collected by the multimodal data collection module S100 to generate intelligent decisions for pick-up point recommendations and route planning.

[0046] Furthermore, the multimodal large model processing module S200 includes: natural language processing NLP, image and video processing, environmental perception and 3D space understanding and language emotion recognition; the natural language processing NLP understands and processes feature text information through the BERT large language model and the method of introducing multi-task learning MTL, and the processing steps are: BERT pre-training, BERT pre-training through a large model autonomous driving corpus, BERT learns a general language representation; fine-tuning BERT and multi-task learning MTL, adding multiple tasks including: classification, proposition entity recognition and question answering to BERT, and optimizing BERT's shared layer and task-specific head; joint loss optimization, through the joint loss function, optimize the goals of different characters and enhance the generalization ability of the model; according to the needs of downstream tasks, the corresponding results are output through the layer corresponding to the task; the BERT pre-training model formula is:

[0047] H t = Transformer(H t－1 , H t－2 , ..., H0; θ)

[0048] Among them, H t represents the hidden state of the Transformer at the tth time step, H t－1 represents the hidden state of the Transformer at the t-1th time step, where the time step refers to the representation of each unit time in a discretized time series; θ represents the model parameters.

[0049] The image and video processing is based on the image and video processing of the multimodal data acquisition module S100, and adds a three-dimensional convolutional neural network 3D-CNN to capture the dynamic characteristics of the target; the steps are: the three-dimensional convolutional neural network 3D-CNN sets a one-dimensional feature extraction structure and a three-dimensional feature extraction structure; obtains the target feature through the one-dimensional feature extraction structure, and extracts the spatiotemporal features of the target that change with time through the three-dimensional feature extraction structure; combines the extracted target features and the corresponding spatiotemporal features to capture the dynamic characteristics of the target.

[0050] The environmental perception and 3D space understanding use LiDAR point cloud processing combined with simultaneous positioning and map construction SLAM technology to generate a real-time 3D environmental map; the steps of generating a real-time 3D environmental map are: extracting local features of point cloud data through LiDAR point cloud processing in sequence through the steps of: acquiring point cloud data, preprocessing point cloud data and extracting features from the preprocessed point cloud data; continuously accumulating point cloud data features and updating positions at different times through map construction SLAM technology to construct a 3D environmental map, and comparing the point cloud obtained by each LiDAR scan with the data of the previous scan to gradually update the 3D environmental map.

[0051] The language emotion recognition refers to classifying speech emotions through an LSTM model based on emotion vectors, and adjusting the pick-up point priority according to the classification results; the emotion recognition model formula is:

[0052] s t =σ(W s ·x t +U s ·h t－1 +b s )

[0053] Among them, s t represents the emotional state, σ represents the Sigmoid activation function, W s and U s Represents the weight parameter of the model, x t represents the input speech feature vector, h t－1 is the hidden state at the previous moment, b s Represents the bias term, which is used to adjust the model output.

[0054] By processing various types of data, different types of data are integrated and analyzed to lay the foundation for subsequent route planning and determination of pick-up points.

[0055] Specifically, for example, there can be interaction between the multimodal data acquisition module and the multimodal large model processing module, interaction between the multimodal large model processing module and the dynamic path planning and pick-up point adjustment module, interaction between the dynamic path planning and pick-up point adjustment module and the personalized pick-up point recommendation module, and interaction between the dynamic path planning and pick-up point adjustment module and the intelligent passenger interaction and feedback module.

[0056] Furthermore, in the interaction between the multimodal data acquisition module and the multimodal large model processing module, the data of the multimodal data acquisition module and the multimodal large model processing module can be fused through the spatiotemporal alignment algorithm to ensure the time synchronization and spatial consistency of the multimodal information; then the multimodal feature fusion is performed; in the interaction between the multimodal large model processing module and the dynamic path planning and pick-up point adjustment module, the path and pick-up points can be planned through path planning based on 3D space understanding, and real-time dynamic adjustment and optimization can be performed; in the dynamic path planning and pick-up point adjustment module and personalized pick-up point recommendation The interaction of modules combines historical data with real-time planning data, combines the real-time updated data of the dynamic route planning and pick-up point adjustment module with the passenger historical data of the personalized pick-up point recommendation module, and generates routes and pick-up points that are closer to passenger needs; in the interaction between the dynamic route planning and pick-up point adjustment module and the intelligent passenger interaction and feedback module, the routes and pick-up points updated in real time by the dynamic route planning and pick-up point adjustment module are combined with the real-time needs of passengers to achieve real-time optimization of routes and pick-up points; the above processes can all be achieved by fusing the extracted features and then obtaining results through corresponding analysis methods.

[0057] In the above embodiment, the personalized pick-up point recommendation module S300 is used to generate personalized pick-up points based on the passenger's historical behavior, preferences and current environment.

[0058] Furthermore, a collaborative filtering algorithm is used to provide customized pick-up point services based on the passenger's historical behavior. The process is to use matrix decomposition to decompose the passenger and pick-up point rating matrix into multiple low-dimensional matrices, so as to determine which pick-up points are more convenient for the passenger. The pick-up point location is determined by the rating value obtained by the inner product of the passenger and pick-up point's feature vectors, and the pick-up point with the highest rating for the passenger's feature vector is selected as the best pick-up point. After the pick-up point is determined, it is adjusted according to environmental changes and passenger needs to ensure the best experience.

[0059] At the same time, the recommended best pick-up point will be marked and other routes and pick-up points with relatively high scores will be recommended to the user. The user will confirm the route and pick-up point within a specific time, such as within 20 seconds; if the user does not confirm, the system will choose the best route and the best pick-up point.

[0060] In the above embodiment, the dynamic path planning and pick-up point adjustment module S400 is used to dynamically adjust the pick-up points and path planning in combination with real-time environmental data and multimodal input; and dynamically adjust the path planning and pick-up points according to the dynamic changes of the real-time environment.

[0061] Furthermore, the dynamic path planning and pick-up point adjustment module S400 includes: real-time environmental data including traffic flow, road closures and weather; dynamic path planning means that the system uses advanced path planning algorithms to select the optimal driving route, and supports dynamic adjustment of pick-up points to ensure that passengers can get on and off the vehicle quickly and safely; and when roads are closed, traffic is congested, and passengers request the location of the pick-up point, the system can adjust the pick-up point location in real time and feed back the adjusted pick-up point location to the passengers; the A* path planning algorithm and reinforcement learning technology are used to ensure that the vehicle can select the optimal path according to changes in the external environment; the A* path planning algorithm is a heuristic path search algorithm used to quickly plan the optimal path under a known map and environment; the system generates the shortest path based on the A* algorithm and combined with real-time traffic data; the A* path planning algorithm formula is:

[0062] f(n)=g(n)+h(n)

[0063] Among them, n represents the node, which represents a certain state and position of the automatic car, g(n) represents the cost from the starting point to the current node, h(n) is the estimated cost from the current node to the target point, and f(n) is the total cost.

[0064] After the optimal path is planned by the A* path planning algorithm, the path needs to be optimized through reinforcement learning. The Q-learning algorithm in deep reinforcement learning DRL is used to achieve real-time path optimization of the vehicle in a dynamic environment. The Q-learning algorithm formula is:

[0065]

[0066] Where Q(s, a) is the state-action value function, which refers to the maximum expected cumulative reward that the autonomous car can obtain by taking action a in state s, r is the immediate reward, γ is a discount factor between 0 and 1, s' refers to the next state to which the autonomous car transfers after taking action a, and a' refers to the possible actions that the autonomous car may take in state s'. It refers to the maximum value of the state-action value function of all possible actions a' in the next state s'.

[0067] In the above embodiment, the intelligent passenger interaction and feedback module S500 is used to interact with passengers in real time, share the driving route and recommended pick-up points with passengers, and passengers view the driving route and recommended pick-up points and propose new requirements based on their needs. The system feeds back the new requirements to the dynamic route planning and pick-up point adjustment module S400, and the dynamic route planning and pick-up point adjustment module S400 adjusts the route and pick-up points according to the new requirements of the passengers.

[0068] Furthermore, the system interacts with passengers through their mobile devices to provide real-time pick-up and drop-off point recommendations and route updates; passengers view recommended pick-up and drop-off points, the vehicle's real-time location, and estimated arrival time through the mobile app; the system also uses augmented reality AR navigation to help passengers quickly find the vehicle's parking location in complex environments; it combines the visual SLAM algorithm with the camera data of the passenger's device to provide accurate AR navigation; the formula for AR navigation is:

[0069] T cw =K[R|t]

[0070] Among them, T cw is the transformation matrix from camera coordinates to world coordinates, and the world coordinates are used to describe the absolute position of objects in the scene; K is the camera intrinsic parameter matrix, R and t are the rotation matrix and translation vector respectively; by determining the absolute positions of objects, passengers and autonomous vehicles in the camera, the position and scene information are mapped to the AR scene, and passengers navigate through the AR rendered interface to find the pick-up point location of the autonomous vehicle.

[0071] like Figure 2 The flowchart of the dynamic pick-up and drop-off of an autonomous driving vehicle based on a multimodal large model according to an embodiment of the present invention; the implementation of the passenger pick-up and drop-off service in the flowchart is based on Figure 1 The system and the analysis method of the system are implemented.

[0072] Specifically, firstly, the voice, text, image, video and sensor data information of the vehicle's surroundings and passengers are collected through the multimodal data collection module; among which, the sensor data refers to LiDAR data, radar data and GPS data, etc.; after the data is collected, it is preprocessed and feature extracted, and the feature-extracted data is sent to the multimodal large model processing module; the multimodal large model processing module performs natural language processing NLP, image and video processing, environmental perception and 3D space understanding and language emotion recognition on the feature-extracted data to understand the deep semantics of different types of data information, and then integrates the deep semantics of different types of data information to generate personalized recommendations for passengers, and send the personalized recommendations to the personalized pick-up point recommendation module; the personalized pick-up point recommendation module has the function of real-time recommendation and adjustment of the pick-up point, and optimization of the pick-up point; the personalized pick-up point recommendation module is based on The passenger's historical pick-up point data information is combined with the environmental data information analyzed by the multimodal large model processing module to recommend personalized pick-up points for passengers, and then the recommended personalized pick-up points are fed back to the dynamic path planning and pick-up point adjustment module; the dynamic path planning and pick-up point adjustment module plans the optimal path and pick-up point in real time based on the recommended personalized pick-up point combined with the real-time fused multimodal data, and feeds back the planned optimal path and pick-up point to the passenger; the passenger views the displayed path and vehicle position, and views the recommended pick-up point through the intelligent passenger interaction and feedback module; if the passenger is not satisfied after viewing, he / she will put forward new requirements, and the mobile phone device used by the passenger will feed back the requirements to the dynamic path planning and pick-up point adjustment module through the passenger app connected to the system, and the dynamic path planning and pick-up point adjustment module will adjust the path and pick-up point according to the passenger's needs; when adjusting the path and pick-up point, the passenger's needs are the first priority.

[0073] At the same time, the dynamic path planning and pick-up point adjustment module not only plans the optimal path and pick-up point, but also performs real-time traffic and environmental perception. It optimizes the generated path by real-time monitoring of traffic and external environmental perception, and feeds the optimized path back to the multimodal large model processing module. The multimodal large model processing module further processes the optimized path in combination with the real-time updated multimodal data to generate a personalized recommended path that meets the passengers' needs.

Claims

1. A dynamic pick-up and drop-off management system for autonomous driving vehicles based on a multimodal large model, characterized in that: The system comprises: Multimodal data acquisition module; Multimodal large model processing module; Personalized pick-up point recommendation module; Dynamic route planning and pick-up point adjustment module; Intelligent passenger interaction and feedback module; The multimodal data acquisition module is used to receive voice, text, image and video input information from passengers, obtain real-time environmental data collected by vehicle sensors, and perform feature extraction on the acquired data; subsequently, the original data and all the data after feature extraction are passed to the multimodal large model processing module; The multimodal large model processing module is used to analyze and fuse the data collected by the multimodal data collection module to generate intelligent decisions for pick-up point recommendation and route planning; The personalized pick-up point recommendation module is used to generate personalized pick-up points based on the passenger's historical behavior, preferences and current environment; The dynamic path planning and pick-up point adjustment module is used to dynamically adjust the pick-up points and path planning in combination with real-time environmental data and multi-modal input; dynamically adjust the path planning and pick-up points according to the dynamic changes of the real-time environment; The intelligent passenger interaction and feedback module is used to interact with passengers in real time, share the driving route and recommended pick-up points with passengers, and passengers view the driving route and recommended pick-up points and propose new requirements based on their needs. The system feeds back the new requirements to the dynamic route planning and pick-up point adjustment module, and the dynamic route planning and pick-up point adjustment module adjusts the route and pick-up points based on the new requirements of the passengers.

2. The multi-modal large model-based dynamic pick-up and drop-off management system for autonomous driving vehicles according to claim 1 is characterized in that: The multimodal data acquisition module includes: data acquisition and data feature extraction; the data acquisition refers to obtaining passenger information and real-time environmental data through various sensors of the self-driving car; the data feature extraction refers to obtaining feature data of the corresponding data by extracting features from the collected data; the data feature extraction is divided into: voice input processing, image and video input processing and sensor data processing; the voice input processing adopts an end-to-end ASR model; the end-to-end ASR model is a speech recognition method that attempts to output text directly from the original voice signal through a unified neural network model, omitting multiple independent steps in the traditional ASR model; that is, the end-to-end ASR model will include an acoustic model, a voice The model and decoder agree on the processing of speech information; the steps of the end-to-end ASR model to process speech information are as follows: pre-training stage: through unsupervised training, the model learns the characteristics of speech signals to generate potential representations, thereby capturing the speech patterns in the audio; fine-tuning stage: using labeled data to further optimize the model so that the model can accurately map from speech signals to text; decoding stage: extracting the final speech-to-text conversion process from the trained model through the Transformer decoder in the model, generating a series of possible text outputs based on the input speech features and the conversion process, and obtaining the final recognition result by selecting the most likely sequence; the speech formula for converting speech signals to text is: Where X represents the speech feature sequence, X = (x1, x2, ..., x T ), T means there are T time steps in X; W means the predicted word sequence, P(W|X) means the conditional probability of predicting the word sequence W given the speech sequence X; x t represents the speech feature at the current time step t, ω t represents the target output at the current time step t, h t－1 represents the hidden state of the previous time step t-1, θ refers to the model parameters, P(ω t |x t ,h t－1 ,θ) indicates the prediction ω at the current time step t t The conditional probability of The image and video input processing adopts feature extraction based on convolutional neural network CNN and visual transformer ViT, and adopts YOLOv5 model to identify the target after feature extraction; the steps of image and video input processing are: preliminarily processing the image and video information through convolutional neural network CNN, extracting low-level features of the image and video information, and the low-level features are local features of the image and video information; the low-level features extracted by the convolutional neural network CNN are transferred to the visual transformer ViT module, and the visual transformer ViT module further captures global information by using the self-attention mechanism to identify long-distance dependencies and context information; combining the features extracted by the convolutional neural network CNN and the visual transformer ViT module, the YOLOv5 model divides the input image into multiple grids according to the combined features, each grid is responsible for predicting the object in the image, and giving the category and bounding box; target recognition is performed by combining the first-order residual structure with multi-scale feature fusion; the first-order residual structure refers to directly adding the input and output of feature extraction by introducing skip connections, so that information can bypass some convolutional layers through these skip connections and be directly transferred to deeper layers; the multi-scale feature fusion adopts feature pyramid network; The sensor data processing combines PointNet++ processing and VoxelNet algorithm for processing; first, point cloud data is obtained by hierarchical data sampling of data in the sensor through PointNet++, and then the point cloud data is converted into voxel Voxel data through the VoxelNet algorithm, and the voxel data is processed through the three-dimensional convolutional neural network 3D CNN technology to obtain low-level local spatial features; finally, PointNet++ aggregates the features extracted from 3DCNN, learns global features, and combines local spatial features with global features to generate 3D structural features of the environment model.

3. The multi-modal large model-based dynamic pick-up and drop-off management system for autonomous driving vehicles according to claim 1, characterized in that: The multimodal large model processing module includes: natural language processing NLP, image and video processing, environmental perception and 3D space understanding and language emotion recognition; the natural language processing NLP understands and processes feature text information through the BERT large language model and the method of introducing multi-task learning MTL, and the processing steps are: BERT pre-training, BERT pre-training through the large model autonomous driving corpus, BERT learns a general language representation; fine-tuning BERT and multi-task learning MTL, adding multiple tasks including: classification, proposition entity recognition and question answering to BERT, and optimizing BERT's shared layer and task-specific head; joint loss optimization, through the joint loss function, optimize the goals of different characters and enhance the generalization ability of the model; according to the needs of downstream tasks, the corresponding results are output through the layer corresponding to the task; the BERT pre-training model formula is: H t =Transformer(H t－1 ,H t－2 ,…,H0;θ) Among them, H t represents the hidden state of the Transformer at the tth time step, H t－1 represents the hidden state of the Transformer at the t-1th time step, where the time step refers to the representation of each unit time in a discretized time series; θ represents the model parameters; The image and video processing is based on the image and video processing of the multimodal data acquisition module, and a three-dimensional convolutional neural network 3D-CNN is added to capture the dynamic characteristics of the target; the steps are: the three-dimensional convolutional neural network 3D-CNN sets a one-dimensional feature extraction structure and a three-dimensional feature extraction structure; the target features are obtained through the one-dimensional feature extraction structure, and the spatiotemporal features of the target that change with time are extracted through the three-dimensional feature extraction structure; the extracted target features and the corresponding spatiotemporal features are combined to capture the dynamic characteristics of the target; The environmental perception and 3D space understanding use LiDAR point cloud processing combined with simultaneous positioning and map construction SLAM technology to generate a real-time 3D environmental map; the steps of generating a real-time 3D environmental map are: extracting local features of point cloud data through LiDAR point cloud processing in sequence through the steps of: acquiring point cloud data, preprocessing point cloud data, and extracting features from the preprocessed point cloud data; continuously accumulating point cloud data features and updating positions at different times through map construction SLAM technology to construct a 3D environmental map, and comparing the point cloud obtained by each LiDAR scan with the data of the previous scan, and gradually updating the 3D environmental map; The language emotion recognition refers to classifying speech emotions through an LSTM model based on emotion vectors, and adjusting the pick-up point priority according to the classification results; the emotion recognition model formula is: s t =σ(W s ·x t +U s ·h t－1 +b s ) Among them, s t represents the emotional state, σ represents the Sigmoid activation function, W s and U s Represents the weight parameter of the model, x t represents the input speech feature vector, h t－1 is the hidden state at the previous moment, b s Represents the bias term, which is used to adjust the model output; By processing various types of data, different types of data are integrated and analyzed to lay the foundation for subsequent route planning and determination of pick-up points.

4. The multi-modal large model-based dynamic pick-up and drop-off management system for autonomous driving vehicles according to claim 1, characterized in that: The personalized pick-up point recommendation module includes: providing customized pick-up point services based on passenger historical behaviors through collaborative filtering algorithms; using matrix decomposition to decompose the passenger and pick-up point rating matrix into multiple low-dimensional matrices, so as to determine which pick-up points are more convenient for passengers; judging the pick-up point location through the rating value obtained by the inner product of the feature vectors of the passenger and the pick-up point, and taking the pick-up point with the highest rating value of the passenger feature vector as the best pick-up point; after the pick-up point is determined, adjusting the pick-up point according to environmental changes and passenger needs to ensure the best experience.

5. The multi-modal large model-based dynamic pick-up and drop-off management system for autonomous driving vehicles according to claim 1, characterized in that: The dynamic path planning and pick-up point adjustment module includes: real-time environmental data including traffic flow, road closures and weather; dynamic path planning means that the system uses advanced path planning algorithms to select the optimal driving route, and supports dynamic adjustment of pick-up points to ensure that passengers can get on and off the vehicle quickly and safely; and when roads are closed, traffic is congested, and passengers request the location of the pick-up point, the system can adjust the pick-up point location in real time and feed back the adjusted pick-up point location to the passengers; the A* path planning algorithm and reinforcement learning technology are used to ensure that the vehicle can select the optimal path according to changes in the external environment; the A* path planning algorithm is a heuristic path search algorithm used to quickly plan the optimal path under a known map and environment; the system generates the shortest path based on the A* algorithm and combined with real-time traffic data; the A* path planning algorithm formula is: f(n)=g(n)+h(n) Among them, n represents a node, which represents a certain state and position of the automatic car, g(n) represents the cost from the starting point to the current node, h(n) is the estimated cost from the current node to the target point, and f(n) is the total cost; After planning the optimal path through the A* path planning algorithm, the path needs to be optimized through reinforcement learning, and the Q-learning algorithm in deep reinforcement learning (DRL) is used to achieve real-time path optimization of the vehicle in a dynamic environment. The Q-learning algorithm formula is: Where Q(s, a) is the state-action value function, which refers to the maximum expected cumulative reward that the autonomous car can obtain by taking action a in state s, r is the immediate reward, γ is a discount factor between 0 and 1, s' refers to the next state to which the autonomous car transfers after taking action a, and a' refers to the possible actions that the autonomous car may take in state s'. It refers to the maximum value of the state-action value function of all possible actions a' in the next state s'.

6. The multi-modal large model-based dynamic pick-up and drop-off management system for autonomous driving vehicles according to claim 1, characterized in that: The intelligent passenger interaction and feedback module includes: the system interacts with passengers through their mobile devices to provide real-time pick-up point recommendations and route updates; passengers view recommended pick-up points, real-time vehicle locations, and estimated arrival times through mobile applications; the system also uses augmented reality (AR) navigation to help passengers quickly find vehicle parking locations in complex environments; combines visual SLAM algorithms with camera data from passenger devices to provide accurate AR navigation; the formula for AR navigation is: T cw =K[Rt] Among them, T cw is the transformation matrix from camera coordinates to world coordinates, and the world coordinates are used to describe the absolute position of objects in the scene; K is the camera intrinsic parameter matrix, R and t are the rotation matrix and translation vector respectively; by determining the absolute positions of objects, passengers and autonomous vehicles in the camera, the position and scene information are mapped to the AR scene, and passengers navigate through the AR rendered interface to find the pick-up point location of the autonomous vehicle.

Images