Multi-modal large model-based system for picking up passengers in last kilometer of automatic driving vehicle

Abstract

The invention provides a system for receiving passengers in the last kilometer of an automatic driving vehicle based on a multi-modal large model, and belongs to the technical field of automatic driving. A multi-modal large model processing module; a path planning and dynamic pick-up point adjusting module; a passenger position prediction and tracking module; a passenger interaction and augmented reality (AR) guiding module; according to the system, passenger data information collection and feature extraction are carried out in a multi-modal data collection and perception module, vehicle surrounding environment perception information analysis is carried out, and various extracted feature information of passengers is fused in a multi-modal large model processing module; a path and a pick-up point generated by combining the vehicle surrounding environment information and the passenger surrounding environment information are more accurate; according to the system, multiple data sources are integrated, pick-up and pick-up point selection, path planning, passenger interaction and the like are intelligently optimized, and it is ensured that pick-up and pick-up tasks of passengers can be efficiently completed in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a last-mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model. Background Art

[0002] In shared travel and self-driving taxi services, the last-mile passenger pick-up problem has always been a key problem that affects user experience. Traditional navigation systems and vehicle path planning often rely on GPS and map information, but GPS signals may be limited in complex urban environments, closed spaces such as buildings and shopping malls, and the system has difficulty accurately identifying the actual location and accessible paths of passengers, resulting in large deviations in the connection between vehicles and passengers. Currently, although self-driving vehicles can navigate to the vicinity of passengers based on preset addresses, in real-world environments, they may not be able to accurately reach the passenger's designated location due to factors such as traffic jams, road closures, and building obstructions. To solve this problem, the new system integrates multimodal large model technology and uses multi-source data from passengers and its own environment to optimize pick-up points and driving routes in real time. Summary of the invention

[0003] The purpose of the present invention is to overcome the above problems existing in the prior art and to greatly improve the technical effect thereof on the basis of the original technology; for this purpose, the present invention provides a last-mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model, the system comprising:

[0004] Multimodal data acquisition and perception module; multimodal large model processing module; path planning and dynamic pick-up point adjustment module; passenger location prediction and tracking module; passenger interaction and augmented reality (AR) guidance module.

[0005] Specifically, the multimodal data acquisition and perception module is used to collect passenger historical data information, data sources provided by passengers, and image and video information around the vehicle itself through various sensors; the multimodal data acquisition and perception module includes: passenger historical data information acquisition channels include cloud, third-party data suppliers, and taxi and shared travel service platforms; the data sources provided by passengers are obtained through the on-board communication module; the data information of the vehicle's own surrounding environment is obtained through various sensors carried by the vehicle itself; the passenger historical data information acquisition includes the passenger's historical order information, historical voice and text information, and the final pick-up point information; the data sources provided by the passenger include: the voice and text information input by the passenger in real time for this taxi service and the image and video information transmitted through the platform APP; the perception module includes: converting voice information into text information, extracting image and video features transmitted by passengers and perception of the vehicle's own surrounding environment; the conversion of voice information into text information includes: converting voice information into text information through an automatic speech recognition ASR system, the steps are: S1, through Wav2Vec The convolutional neural network CNN in the 2.0 model extracts the features of the passenger's voice information and uses the Transformer network to perform context modeling on the extracted features. The output is a high-dimensional feature representation sequence. S2. Perform CTC decoding: The deep neural network LSTM receives the information from Wav2Vec 2.0 context feature representation sequence, and trained by CTC loss function; CTC loss function reduces the requirement of time alignment by dealing with the situation that the length of input sequence is not aligned with the length of output text, so that the model can learn without explicit alignment; CTC decoder generates final text output based on the predicted probability by predicting the probability distribution of characters at each moment; S3, finally, the characters output by the model are processed through post-processing steps including removing repeated characters and blanks to generate the final text result; the extraction of image and video features transmitted by passengers includes: feature extraction by three-dimensional convolutional neural network 3D CNN and YOLOv5, the steps are: S11, converting video information into video frame information, extracting features of video frame information by three-dimensional convolutional neural network 3D CNN, and extracting spatiotemporal features and structural features in the image information provided by passengers; S12, extracting 3D The features extracted by CNN are used to segment and detect the targets in the image, and the targets in the image information are identified; the perception of the vehicle's own surrounding environment includes: the autonomous driving vehicle's perception of its own surrounding environment, and the perception steps are: S21, preprocessing the acquired external image information and video information; S22, inputting the preprocessed information into the combination of convolutional neural network CNN and YOLOv5 to perform target detection, classification and segmentation of images and videos; S23, after the target recognition and classification are completed, predicting the current environment of the vehicle through the following: context understanding, environmental perception and prediction of target behavior and intention; wherein, predicting the current environment of the vehicle includes: using extended Kalman filter EKF to fuse the data of various sensors including: camera, radar and LiDAR and the extracted image features to generate real-time information about the surrounding environment of the vehicle; the Kalman filter formula is:.

[0006] x k =Ax k－1 +Bu k +ω k

[0007] Among them, x k represents the system state at the current time k, x k－1 represents the state at the previous moment, A represents the state matrix from time k-1 to time k, B represents the influence matrix of control input on the state, u k represents the control input at the current moment, ω k is the process noise.

[0008] Specifically, the multimodal large model processing module is used to fuse the collected multiple data through the multimodal large model technology to help the system understand the pick-up and drop-off requirements in complex scenarios; the multimodal large model processing module includes: natural language processing NLP, image and video analysis, and 3D space perception and understanding; the natural language processing is used to parse the text information entered by the passenger and the extracted feature text information, extract the location information in the passenger description, and determine the approximate pick-up and drop-off area in combination with the map data; the image and video analysis uses image and video analysis technology to identify the environmental information uploaded by the passenger through the features of the extracted image and video information, and further confirm the buildings, signs and other key reference objects around the pick-up point; the environmental perception and 3D space analysis The solution is used to generate a 3D model of the surrounding environment through multimodal sensors, especially in complex urban environments and multi-story buildings, the system can perceive the precise spatial relationship between the vehicle and the passengers; the natural language processing NLP uses the BERT model to process the received text, giving priority to the text information converted from voice information; the BERT model can understand the location details described by the passenger through bidirectional context encoding, and preliminarily generate the passenger's location information in combination with map data; then, through the self-attention Attention mechanism in the BERT model, the location clues in the text are matched with the environmental image and video data to realize the recognition of the actual location of the passenger; the formula of the Attention mechanism is:

[0009]

[0010] Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix.

[0011] The image and video analysis includes: applying a multimodal large model based on ViT to fuse the extracted image and video features and text features to generate a comprehensive understanding of the passenger's environment; the steps of generating a comprehensive understanding of the passenger's environment are: S31, fusing the obtained image and video features and text features through early fusion, mid-term fusion and late fusion of ViT's multimodal large model technology; S32, after the features are fused, ViT's multimodal large model will make further processing of environmental understanding and behavioral decisions based on the fused features to generate a comprehensive understanding of the passenger's environment; S33, optimizing model training: by adopting a joint training method to simultaneously optimize the image and text feature extractors to ensure that the two modalities can work together; through multimodal data input, that is, inputting image features and text features to optimize the model's performance in both image understanding and text understanding, thereby improving the overall decision-making ability of the system; S34, after the model training is completed, the real-time extracted image features and text features are input into the trained model, and the passenger environment information is obtained based on the output.

[0012] The 3D space perception and understanding includes: combining the obtained position information and the passenger's environment information, and using the BEV Transformer algorithm to convert the 2D data obtained by the vehicle sensors, including the camera and LiDAR, into a 3D space representation, using the Transformer's self-attention mechanism to flexibly model the three-dimensional spatial relationship in complex scenes, and generate an accurate three-dimensional space model of the environment around the pick-up point; at the same time, introducing the Occupancy Network model to predict the occupancy of the generated three-dimensional space, and combining the dynamically changing environment for space perception and prediction; the formula for generating the space representation using the BE V Transformer is:

[0013] S 3D =g(S 2D , D depth )

[0014] Among them, S 3D is the 3D space representation, S 2D It is 2D sensor data, which refers to image information and video frame information. depth is depth information, g() is a mapping function, and the mapping function is usually implemented by a neural network.

[0015] Specifically, the path planning and dynamic pick-up point adjustment module is used to generate the optimal path planning plan according to the real-time perceived multimodal data, and dynamically adjust the pick-up point when necessary; the path planning and dynamic pick-up point adjustment module includes: path planning and dynamic adjustment and feedback; the path planning includes: combining A* and Dijkstra algorithms to search for the optimal path, and the search steps are: S41. First, set the actual cost g(n) from the self-driving car to all nodes has not been calculated, set the h(n) value to the heuristic estimated distance from node n to the passenger, and set the starting point to (g(start), h(start)) as the estimated distance from the starting point to the target; finally, initialize a priority queue for processing nodes in the sequence of f(n)=g(n)+h(n); S42. Add the starting point to the priority queue; take the node with the smallest current f(n) value from the queue as the current node; update all adjacent nodes of the current node: first calculate the distance from the current node to The actual path cost g(n) of the adjacent node, calculate the heuristic estimate h(n) of the adjacent node, calculate the total evaluation value f(n) = g(n) + h(n) of the adjacent node, if the corresponding adjacent node has not been visited and a shorter path can be found through the current node, then update the g, h, f values ​​of the adjacent node, add the corresponding adjacent node to the priority queue, and record the predecessor node of the adjacent node as the current node at this time; S43, if the current node is the target node, stop searching and rebuild the path through the recorded predecessor nodes; when updating the current node, the predecessor node of each node is recorded, therefore, by backtracking, start from the target node until returning to the starting point; S44, if the target node is found and the optimal path is generated, terminate the search and output the shortest path from the starting point to the target, that is, the optimal path; if the target node is not found through the current node, update the current node through the updated priority queue, and go through S42 and S43 until the target node is found.

[0016] Furthermore, the dynamic adjustment and feedback include: dynamically adjusting and optimizing the obtained path through the deep reinforcement learning DRL model and the inverse reinforcement learning IRL model; wherein, the path is dynamically adjusted through the deep reinforcement learning DRL model, and the self-driving car is trained to flexibly respond to road closures and emergencies in complex environments through algorithms; then, the inverse reinforcement learning IRL model is applied to further improve the path selection of the self-driving car by learning human driving behavior, and the inverse reinforcement learning IRL model updates the pick-up point in real time by considering the passenger waiting time and road congestion to ensure that the vehicle reaches the passenger in the shortest time. At the same time, during the process of picking up and dropping off passengers, if the autonomous driving car determines through the path that the distance to the passenger is less than 500 meters, the message of the impending arrival will be fed back to the passenger by sending information and vibrating to the passenger's device, making the pick-up and drop-off service more efficient; if the autonomous driving car determines through the path that the distance to the passenger is less than 500 meters, and there is no pick-up point in the passenger's area due to the passenger's change of position, the passenger's location will be fed back to the passenger by sending information to the passenger's device, and a pick-up point will be set up nearby for the passenger to choose. After the passenger makes his choice, the path and pick-up point will be adjusted according to the pick-up point selected by the passenger.

[0017] Specifically, the passenger location prediction and tracking module tracks the movement of passengers through positioning and tracking devices, and predicts the possible location changes of passengers by combining the collected historical behavior data of passengers with real-time data; the passenger location prediction and tracking module includes: passenger location tracking and passenger behavior prediction; passenger location tracking includes: accurate positioning of passengers through multi-source positioning technologies including GPS, WiFi and Bluetooth; then, the deep learning time series model LSTM is used to analyze the passenger's movement pattern, predict the possible movement direction and location changes in the future, and adjust the pick-up point in advance; finally, dynamic tracking is performed by combining the passenger's historical positioning data with the particle filter algorithm to ensure that the vehicle's path is adjusted in time when the passenger moves to another pick-up point; passenger behavior prediction includes: the behavior prediction model based on the dynamic Bayesian network DBN combines the passenger's real-time location and historical behavior data to make a probability prediction of the next location; through model updating, the speed and direction of passenger movement are dynamically adapted; among them, the prediction of passenger location based on the dynamic Bayesian network DBN is based on Bayesian estimation, and the passenger location prediction formula based on Bayesian estimation is:

[0018]

[0019] Among them, X t is the passenger position at time t, X t－1 is the passenger position at time t-1, O t is the sensor observation data at time t, P(X t X t－1 , O t) represents the posterior probability, P(O t X t ) represents the observation probability, P(X t X t－1 ) represents the state transition probability, P(O t ) represents the marginal probability of observation.

[0020] Specifically, the passenger interaction and augmented reality AR guidance module interacts with passengers through the passenger smartphone application, provides real-time pick-up point information and path updates through interaction, and helps passengers find the correct boarding location through augmented reality technology AR; the passenger interaction and augmented reality AR guidance module includes: real-time information push and AR navigation guidance; the real-time information push includes: using the WebSocket protocol to achieve low-latency information push through the connection with the autonomous driving app in the passenger's mobile device, ensuring that the passenger can obtain the vehicle location, pick-up point information and path adjustment in real time; at the same time, receiving the information feedback from the passenger through the app, and adjusting the path and pick-up point in time according to the information feedback from the passenger; the AR navigation guidance includes: constructing a 3D model of the passenger's surrounding environment through an augmented reality system based on SLAM technology combined with the acquired passenger environment information, and using the Unity-based AR framework to generate intuitive visual guidance to help passengers find the correct boarding location; through real-time information push and AR navigation, the path and pick-up point of the autonomous driving car are dynamically adjusted, and at the same time, passengers can accurately reach the pick-up point location, improve the pick-up efficiency of the autonomous driving car, and increase the passenger's riding experience.

[0021] The beneficial effects of the present invention are:

[0022] The present invention provides a last-mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model, the system comprising a multimodal data acquisition and perception module; a multimodal large model processing module; a path planning and dynamic pick-up point adjustment module; a passenger position prediction and tracking module; a passenger interaction and augmented reality (AR) guidance module; the system collects and extracts passenger data information in the multimodal data acquisition and perception module, and analyzes vehicle surrounding environment perception information; in the multimodal large model processing module, various feature information of the extracted passengers is integrated, and the spatial environment information between the autonomous driving vehicle and the passenger is generated in combination with the vehicle surrounding environment perception information, and the spatial environment information between the autonomous driving vehicle and the passenger generated in this way will be more accurate; in the path planning and dynamic pick-up point adjustment module, the A* and Dijkstra algorithms are combined to search for the optimal path, and the advantages of the two algorithms are combined to search for the optimal path, and the optimal path obtained is more reliable; finally, the system integrates multiple data sources, intelligently optimizes the pick-up point selection, path planning and passenger interaction, etc., to ensure that the passenger pick-up task can be efficiently completed in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the last-mile passenger picking-up system of an autonomous driving vehicle based on a multimodal large model of the present invention.

[0024] Figure 2 This is a flow chart of the last mile passenger picking up by an autonomous driving vehicle based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0025] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention, and cannot be used to limit the present invention.

[0026] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0027] like Figure 1 This is a schematic diagram of a last-mile passenger pick-up system for an autonomous driving vehicle based on a multimodal large model according to an embodiment of the present invention, which includes: a multimodal data acquisition and perception module S100; a multimodal large model processing module S200; a path planning and dynamic pick-up point adjustment module S300; a passenger location prediction and tracking module S400; and a passenger interaction and augmented reality (AR) guidance module S500.

[0028] Specifically, the multimodal data acquisition and perception module S100 is used to collect passenger historical data information, data sources provided by passengers, and image and video information around the vehicle itself through various sensors; the multimodal data acquisition and perception module S100 includes: the passenger historical data information acquisition channels include cloud, third-party data suppliers, and taxi and shared travel service platforms; the data sources provided by passengers are obtained through the on-board communication module; the data information of the vehicle's own surrounding environment is obtained through various sensors carried by the vehicle itself; the passenger historical data information acquisition includes the passenger's historical order information, historical voice and text information, and the final pick-up point information; the data sources provided by the passenger include: the voice and text information input by the passenger in real time for this taxi service and the image and video information transmitted through the platform APP; the perception module includes: converting voice information into text information, extracting image and video features transmitted by passengers and perception of the vehicle's own surrounding environment; the converting of voice information into text information includes: converting voice information into text information through an automatic speech recognition ASR system, the steps are: S1, through Wav2Vec The convolutional neural network CNN in the 2.0 model extracts the features of the passenger's voice information, and uses the Transformer network to perform context modeling on the extracted features, and the output is a high-dimensional feature representation sequence; S2, CTC decoding: the deep neural network LSTM receives the context feature representation sequence from Wav2Vec 2.0 and is trained through the CTC loss function; the CTC loss function reduces the requirement for time alignment by dealing with the situation where the length of the input sequence is not aligned with the length of the output text, so that the model can learn without explicit alignment; the CTC decoder predicts the probability distribution of characters at each moment and generates the final text output based on the predicted probability; S3, finally, the characters output by the model undergo post-processing steps including removing repeated characters and blanks to generate the final text result; the extraction of image and video features transmitted by passengers includes: feature extraction through three-dimensional convolutional neural network 3D CNN and YOLOv5, the steps are: S11, converting video information into video frame information, extracting features of the video frame information through the three-dimensional convolutional neural network 3D CNN, and extracting the spatiotemporal features and structural features in the image information provided by the passenger; S12, 3D The features extracted by CNN are used to segment and detect targets in the image, and targets in the image information are identified; the perception of the vehicle's own surrounding environment includes: the autonomous driving vehicle's perception of its own surrounding environment, and the perception steps are: S21, preprocessing the acquired external image information and video information; S22, inputting the preprocessed information into a combination of convolutional neural network CNN and YOLOv5 to perform target detection, classification and segmentation of images and videos;

[0029] S23. After the target is identified and classified, the environment in which the vehicle is currently located is predicted by: context understanding, environmental perception, and prediction of the target's behavior and intention; wherein, predicting the environment in which the vehicle is currently located includes: using an extended Kalman filter (EKF) to fuse data from various sensors including cameras, radars, and LiDARs and extracted image features to generate real-time information about the vehicle's surrounding environment; the Kalman filter formula is:

[0030] x k =Ax k－1 +Bu k +ω k

[0031] Among them, x k represents the system state at the current time k, x k－1 represents the state at the previous moment, A represents the state matrix from time k-1 to time k, B represents the influence matrix of control input on the state, u k represents the control input at the current moment, ω k is the process noise.

[0032] Specifically, the multimodal large model processing module S200 is used to fuse and process the collected multiple data through the multimodal large model technology to help the system understand the pick-up and drop-off requirements in complex scenarios; the multimodal large model processing module S200 includes: natural language processing NLP, image and video analysis, and 3D space perception and understanding; the natural language processing is used to parse the text information input by the passenger and the extracted feature text information, extract the location information in the passenger description, and determine the approximate pick-up and drop-off area in combination with the map data; the image and video analysis uses image and video analysis technology to identify the environmental information uploaded by the passenger through the features of the extracted image and video information, and further confirm the buildings, signs and other key reference objects around the pick-up point; the environmental perception and 3D spatial perception and understanding are used to analyze ... 3D spatial understanding is used to generate a 3D model of the surrounding environment through multimodal sensors, especially in complex urban environments and multi-story buildings, the system can perceive the precise spatial relationship between the vehicle and the passenger; the natural language processing NLP uses the BERT model to process the received text, giving priority to the text information converted from voice information; the BERT model can understand the location details described by the passenger through bidirectional context encoding, and preliminarily generate the passenger's location information in combination with map data; further, through the self-attention Attention mechanism in the BERT model, the location clues in the text are matched with the environmental image and video data to realize the recognition of the actual location of the passenger; the formula of the Attention mechanism is:

[0033]

[0034] Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix;

[0035] The image and video analysis includes: applying a multimodal large model based on ViT to fuse the extracted image and video features and text features to generate a comprehensive understanding of the passenger's environment; the steps of generating a comprehensive understanding of the passenger's environment are: S31, fusing the obtained image and video features and text features through early fusion, mid-term fusion and late fusion of ViT's multimodal large model technology; S32, after the features are fused, ViT's multimodal large model will make further processing of environmental understanding and behavioral decisions based on the fused features to generate a comprehensive understanding of the passenger's environment; S33, optimizing model training: by adopting a joint training method to simultaneously optimize the image and text feature extractors to ensure that the two modalities can work together; through multimodal data input, that is, inputting image features and text features to optimize the model's performance in both image understanding and text understanding, thereby improving the overall decision-making ability of the system; S34, after the model training is completed, the real-time extracted image features and text features are input into the trained model, and the passenger environment information is obtained based on the output.

[0036] The 3D space perception and understanding includes: combining the obtained position information and the passenger's environment information, and using the BEV Transformer algorithm to convert the 2D data obtained by the vehicle sensors, including the camera and LiDAR, into a 3D space representation, using the Transformer's self-attention mechanism to flexibly model the three-dimensional spatial relationship in complex scenes, and generate an accurate three-dimensional space model of the environment around the pick-up point; at the same time, introducing the Occupancy Network model to predict the occupancy of the generated three-dimensional space, and combining the dynamically changing environment for space perception and prediction; the formula for generating the space representation using the BE V Transformer is:

[0037] S 3D =g(S 2D , D depth )

[0038] Among them, S 3D is the 3D space representation, S 2D It is 2D sensor data, which refers to image information and video frame information. depth is depth information, g() is a mapping function, and the mapping function is usually implemented by a neural network.

[0039] Specifically, the path planning and dynamic pick-up point adjustment module S300 is used to generate the optimal path planning plan according to the real-time perceived multimodal data, and dynamically adjust the pick-up point when necessary; the path planning and dynamic pick-up point adjustment module S300 includes: path planning and dynamic adjustment and feedback; the path planning includes: combining A* and Dijkstra algorithms to search for the optimal path, and the search steps are: S41. First, set the actual cost g(n) from the self-driving car to all nodes has not been calculated, set the h(n) value to the heuristic estimated distance from node n to the passenger, and set the starting point to (g(start), h(start)) as the estimated distance from the starting point to the target; finally, initialize a priority queue for processing nodes in the sequence of f(n)=g(n)+h(n); S42. Add the starting point to the priority queue; take the node with the smallest current f(n) value from the queue as the current node; update all adjacent nodes of the current node: first calculate the current The actual path cost g(n) from the node to the adjacent node, calculate the heuristic estimate h(n) of the adjacent node, calculate the total evaluation value f(n) = g(n) + h(n) of the adjacent node, if the corresponding adjacent node has not been visited and a shorter path can be found through the current node, then update the g, h, f values ​​of the adjacent node, and add the corresponding adjacent node to the priority queue, and record the predecessor node of the adjacent node as the current node at this time; S43, if the current node is the target node, stop searching and rebuild the path through the recorded predecessor nodes; when updating the current node, the predecessor node of each node is recorded, therefore, by backtracking, start from the target node until returning to the starting point; S44, if the target node is found and the optimal path is generated, terminate the search and output the shortest path from the starting point to the target, that is, the optimal path; if the target node is not found through the current node, update the current node through the updated priority queue, and go through S42 and S43 until the target node is found.

[0040] The dynamic adjustment and feedback include: dynamically adjusting and optimizing the obtained path through a deep reinforcement learning DRL model and an inverse reinforcement learning IRL model; wherein, the path is dynamically adjusted through a deep reinforcement learning DRL model, and the self-driving car is trained by an algorithm to flexibly respond to road closures and emergencies in a complex environment; subsequently, the inverse reinforcement learning IRL model is applied to further improve the path selection of the self-driving car by learning human driving behavior, and the inverse reinforcement learning IRL model updates the pick-up point in real time by considering the waiting time of passengers and road congestion, ensuring that the vehicle reaches the passenger's location in the shortest time; at the same time, during the process of picking up and dropping off passengers, if the self-driving car judges that the distance from the passenger is less than 500 meters through the path, the message of the upcoming arrival is fed back to the passenger by sending information and vibrating to the passenger's device, so as to make the pick-up service more efficient; if the self-driving car judges that the distance from the passenger is less than 500 meters through the path, and there is no pick-up point in the passenger's area due to the passenger's change of position, the passenger's location is fed back to the passenger by sending information to the passenger's device, and a pick-up point is set nearby for the passenger to choose, and after the passenger chooses, the path and the pick-up point are adjusted according to the pick-up point selected by the passenger.

[0041] Specifically, the passenger location prediction and tracking module S400 tracks the movement of passengers through the positioning tracking device, and predicts the possible location changes of passengers by combining the collected historical behavior data of passengers with real-time data; the passenger location prediction and tracking module S400 includes: passenger location tracking and passenger behavior prediction; the passenger location tracking includes: accurate positioning of passengers through multi-source positioning technology including GPS, WiFi and Bluetooth; then, using the deep learning time series model LSTM to analyze the passenger's movement pattern, predict the possible future movement direction and location changes, and adjust the pick-up point in advance; finally, through the particle filter algorithm combined with the passenger's historical positioning data for dynamic tracking, ensure that when the passenger moves to another pick-up point, the vehicle's path is adjusted in time; the passenger behavior prediction includes: based on the behavior prediction model of the dynamic Bayesian network DBN combined with the real-time location and historical behavior data of the passenger, the probability prediction of the next location is made; through model updating, the speed and direction of the passenger's movement are dynamically adapted; wherein, the prediction of the passenger location based on the dynamic Bayesian network DBN is based on Bayesian estimation, and the passenger location prediction formula based on Bayesian estimation is:

[0042]

[0043] Among them, X t is the passenger position at time t, X t－1 is the passenger position at time t-1, O t is the sensor observation data at time t, P(X t X t－1 , Ot ) represents the posterior probability, P(O t X t ) represents the observation probability, P(X t X t－1 ) represents the state transition probability, P(O t ) represents the marginal probability of observation.

[0044] Specifically, the passenger interaction and augmented reality AR guidance module S500 interacts with passengers through the passenger smartphone application, provides real-time pick-up point information and path updates through interaction, and helps passengers find the correct boarding location through augmented reality technology AR; the passenger interaction and augmented reality AR guidance module S500 includes: real-time information push and AR navigation guidance; the real-time information push includes: using the WebSocket protocol to achieve low-latency information push through the connection with the autonomous driving app in the passenger's mobile device, ensuring that the passenger can obtain the vehicle location, pick-up point information and path adjustment in real time; at the same time, receiving the information feedback from the passenger through the app, and adjusting the path and pick-up point in time according to the information feedback from the passenger; the AR navigation guidance includes: constructing a 3D model of the passenger's surrounding environment through an augmented reality system based on SLAM technology combined with the acquired passenger environment information, and using the Unity-based AR framework to generate intuitive visual guidance to help passengers find the correct boarding location; through real-time information push and AR navigation, the path and pick-up point of the autonomous driving car are dynamically adjusted, and at the same time, the passengers can accurately reach the pick-up point location, improve the pick-up efficiency of the autonomous driving car, and increase the passenger's riding experience.

[0045] like Figure 2 This is a flowchart of the last mile passenger pick-up of an autonomous driving vehicle based on a multimodal large model according to an embodiment of the present invention; the implementation of the passenger pick-up service in the flowchart is based on Figure 1 The process of the flowchart is roughly as follows: first, the multimodal data acquisition and perception module S100 of the system collects data source information provided by passengers, historical information of passengers riding, environmental information around the vehicle, and positioning information; and the perception module extracts features of the collected passenger data information, performs environmental perception processing on the environmental information around the vehicle, and connects the environment around the vehicle; the extracted passenger data information features and the perceived environmental information around the vehicle are transmitted to the multimodal large model processing module S200, and the multimodal large model processing module S200 is processed as follows Figure 1 The method described above performs feature fusion on the feature data, and sends the fused data information and the environmental information around the vehicle to the path planning and dynamic pick-up point adjustment module S300; the path planning and dynamic pick-up point adjustment module S300 analyzes the fused data analysis results and the environmental information around the vehicle according to Figure 1The above method is used to perform path planning and pick-up point determination, and the determined path information and pick-up point information are sent to the passenger location prediction and tracking module S400; the passenger location prediction and tracking module S400 is based on Figure 1 The technology described in the above method locates, tracks and predicts the passenger's position; and dynamically adjusts the pick-up point in real time in combination with path planning, and transmits the real-time generated dynamic pick-up point adjustment and the generated path to the passenger interaction and augmented reality AR guidance module S500 in real time; the passenger interaction and augmented reality AR guidance module S500 pushes the latest path adjustment, distance information and pick-up point information to the passenger in real time. At the same time, the augmented reality system based on SLAM technology is combined with Unity's AR framework to generate intuitive visual guidance, guiding passengers to quickly reach the vehicle's pick-up point through the autonomous driving app of the mobile device, thereby improving the pick-up efficiency of the autonomous driving vehicle and increasing the passenger's riding experience.

Claims

1. The last mile passenger pickup system of an autonomous driving vehicle based on a multimodal large model is characterized by: The system comprises: Multimodal data collection and perception module: used to collect passenger historical data information, data sources provided by passengers, and image and video information around the vehicle itself through various sensors; Multimodal large model processing module: used to integrate and process various collected data through multimodal large model technology to help the system understand the pick-up and drop-off requirements in complex scenarios; Path planning and dynamic pick-up point adjustment module: used to generate the optimal path planning plan based on real-time multimodal data, and dynamically adjust the pick-up points when necessary; Passenger location prediction and tracking module: Track the movement of passengers through positioning tracking devices, and predict the possible location changes of passengers by combining the collected historical behavior data of passengers with real-time data; Passenger interaction and augmented reality (AR) guidance module: The system interacts with passengers through the passenger smartphone application, provides real-time pick-up point information and route updates through interaction, and helps passengers find the correct boarding location through augmented reality technology (AR).

2. The last mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model according to claim 1, characterized in that: The multimodal data collection includes: the passenger historical data information is obtained through the cloud, third-party data suppliers and taxi and shared travel service platforms; the data source provided by the passenger is obtained through the vehicle communication module; the data information of the vehicle's own surrounding environment is obtained through various sensors carried by the vehicle itself; the passenger historical data information acquisition includes the passenger's historical order information, historical voice text information and the final pick-up point information; the data source provided by the passenger includes: the voice text information input by the passenger in real time for this taxi service and the image and video information transmitted through the platform APP; the perception module includes: converting voice information into text information, extracting the image and video features transmitted by the passenger and the perception of the vehicle's own surrounding environment; the voice information is converted into text information including: converting voice information into text information through the automatic speech recognition ASR system, the steps are: S1, extracting the features of the passenger's voice information through the convolutional neural network CNN in the Wa v2Vec 2.0 model, and using the Transformer network to perform context modeling on the extracted features, and the output is a high-dimensional feature representation sequence; S2, performing CTC decoding: the deep neural network LSTM receives the data from the Wa v2Vec 2.0 model.0 context feature representation sequence, and trained by CTC loss function; CTC loss function reduces the requirement of time alignment by dealing with the situation that the length of input sequence is not aligned with the length of output text, so that the model can learn without explicit alignment; CTC decoder generates final text output based on the predicted probability by predicting the probability distribution of characters at each moment; S3, finally, the characters output by the model are processed through post-processing steps including removing repeated characters and blanks to generate the final text result; the extraction of image and video features transmitted by passengers includes: feature extraction by three-dimensional convolutional neural network 3D CNN and YOLOv5, the steps are: S11, converting video information into video frame information, extracting features of video frame information by three-dimensional convolutional neural network 3D CNN, and extracting spatiotemporal features and structural features in the image information provided by passengers; S12, extracting 3D The features extracted by CNN are used to segment and detect the targets in the image, and the targets in the image information are identified; the perception of the vehicle's own surrounding environment includes: the autonomous driving vehicle's perception of its own surrounding environment, and the perception steps are: S21, preprocessing the acquired external image information and video information; S22, inputting the preprocessed information into the combination of convolutional neural network CNN and YOLOv5 to perform target detection, classification and segmentation of images and videos; S23, after the target recognition and classification are completed, predicting the current environment of the vehicle through the following: context understanding, environmental perception and prediction of target behavior and intention; wherein, predicting the current environment of the vehicle includes: using extended Kalman filter EKF to fuse the data of various sensors including: camera, radar and LiDAR and the extracted image features to generate real-time information about the surrounding environment of the vehicle; the Kalman filter formula is:. x k =Ax k－1 +Bu k +ω k Among them, x k represents the system state at the current time k, x k－1 represents the state at the previous moment, A represents the state matrix from time k-1 to time k, B represents the influence matrix of control input on the state, u k represents the control input at the current moment, ω k is the process noise.

3. The last mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model according to claim 1, characterized in that: The multimodal large model processing module includes: natural language processing NLP, image and video analysis, and 3D space perception and understanding; the natural language processing is used to parse the text information input by the passenger and the extracted feature text information, extract the location information in the passenger description, and determine the approximate pick-up area in combination with the map data; the image and video analysis uses image and video analysis technology to identify the environmental information uploaded by the passenger through the features of the extracted image and video information, and further confirm the buildings, signs and other key reference objects around the pick-up point; the environmental perception and 3D space understanding are used to generate a 3D model of the surrounding environment through multimodal sensors, especially in complex urban environments and multi-story buildings, the system can perceive the precise spatial relationship between the vehicle and the passenger; the natural language processing NLP uses the BERT model to process the received text, giving priority to the text information converted from voice information; the BERT model can understand the location details described by the passenger through bidirectional context encoding, and preliminarily generate the passenger's location information in combination with the map data; then, through the self-attention mechanism in the BERT model, the location clues in the text are matched with the environmental image and video data to realize the recognition of the actual location of the passenger; The formula of the Attention mechanism is: Among them, Q, K and V are query vector, key vector and value vector respectively, d k is the dimension of the key vector, Attention is the self-attention function, softmax is the probability distribution function used to convert the attention score into a probability distribution, and T is the transpose of the matrix; The image and video analysis includes: applying a multimodal large model based on ViT to fuse the extracted image and video features and text features to generate a comprehensive understanding of the passenger's environment; the steps of generating a comprehensive understanding of the passenger's environment are: S31, fusing the obtained image and video features and text features through early fusion, mid-term fusion and late fusion of ViT's multimodal large model technology; S32, after the features are fused, ViT's multimodal large model will make further processing of environmental understanding and behavioral decisions based on the fused features to generate a comprehensive understanding of the passenger's environment; S33, optimizing the model training: optimizing the image and text feature extractors at the same time by adopting a joint training method to ensure that the two modalities can work together; optimizing the model's performance in image understanding and text understanding through multimodal data input, that is, inputting image features and text features, thereby improving the overall decision-making ability of the system; S34, after the model training is completed, inputting the real-time extracted image features and text features into the trained model, and obtaining the passenger environment information based on the output; The 3D space perception and understanding includes: combining the obtained position information and the passenger's environment information, and using the BEV Transformer algorithm to convert the 2D data obtained by the vehicle sensors, including the camera and LiDAR, into a 3D space representation, using the Transformer's self-attention mechanism to flexibly model the three-dimensional spatial relationship in complex scenes, and generate an accurate three-dimensional space model of the environment around the pick-up point; at the same time, introducing the Occupancy Network model to predict the occupancy of the generated three-dimensional space, and combining the dynamically changing environment for spatial perception and prediction; the formula for generating the spatial representation using the BEV Transformer is: S 3D =g(S 2D ,D depth ) Among them, S 3D is the 3D space representation, S 2D It is 2D sensor data, which refers to image information and video frame information. depth is depth information, g() is a mapping function, and the mapping function is usually implemented by a neural network.

4. The last mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model according to claim 1, characterized in that: The path planning and dynamic pick-up point adjustment module includes: path planning and dynamic adjustment and feedback; the path planning includes: combining A* and Dijkstra algorithms to search for the optimal path, and the search steps are: S41. First, set the actual cost g(n) from the self-driving car to all nodes has not been calculated, set the h(n) value to the heuristic estimated distance from node n to the passenger, and set the starting point to (g(start), h(start)) as the estimated distance from the starting point to the target; finally, initialize a priority queue for processing nodes in the sequence of f(n)=g(n)+h(n); S42. Add the starting point to the priority queue; take the node with the smallest current f(n) value from the queue as the current node; update all adjacent nodes of the current node: first calculate the actual path cost g(n) from the current node to the adjacent node, calculate the heuristic estimate of the adjacent node Value h(n), calculate the total evaluation value f(n) = g(n) + h(n) of the adjacent node. If the corresponding adjacent node has not been visited and a shorter path can be found through the current node, update the g, h, and f values ​​of the adjacent node, add the corresponding adjacent node to the priority queue, and record that the predecessor node of the adjacent node is the current node at this time; S43, if the current node is the target node, stop searching and rebuild the path through the recorded predecessor node; when updating the current node, the predecessor node of each node is recorded, so by backtracking, start from the target node until you return to the starting point; S44, if the target node is found and the optimal path is generated, terminate the search and output the shortest path from the starting point to the target, that is, the optimal path; if the target node is not found through the current node, update the current node through the updated priority queue, and go through S42 and S43 until the target node is found; The dynamic adjustment and feedback include: dynamically adjusting and optimizing the obtained path through a deep reinforcement learning DRL model and an inverse reinforcement learning IRL model; wherein, the path is dynamically adjusted through a deep reinforcement learning DRL model, and the self-driving car is trained by an algorithm to flexibly respond to road closures and emergencies in a complex environment; subsequently, the inverse reinforcement learning IRL model is applied to further improve the path selection of the self-driving car by learning human driving behavior, and the inverse reinforcement learning IRL model updates the pick-up point in real time by considering the waiting time of passengers and road congestion, ensuring that the vehicle reaches the passenger's location in the shortest time; at the same time, during the process of picking up and dropping off passengers, if the self-driving car judges that the distance from the passenger is less than 500 meters through the path, the message of the upcoming arrival is fed back to the passenger by sending information and vibrating to the passenger's device, so as to make the pick-up service more efficient; if the self-driving car judges that the distance from the passenger is less than 500 meters through the path, and there is no pick-up point in the passenger's area due to the passenger's change of position, the passenger's location is fed back to the passenger by sending information to the passenger's device, and a pick-up point is set nearby for the passenger to choose, and after the passenger chooses, the path and the pick-up point are adjusted according to the pick-up point selected by the passenger.

5. The last mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model according to claim 1, characterized in that: The passenger location prediction and tracking module includes: passenger location tracking and passenger behavior prediction; the passenger location tracking includes: accurate positioning of passengers through multi-source positioning technology including GPS, WiFi and Bluetooth; then, using the deep learning time series model LSTM to analyze the passenger's movement pattern, predict the possible future movement direction and position changes, and adjust the pick-up point in advance; finally, through the particle filter algorithm combined with the passenger's historical positioning data for dynamic tracking, to ensure that when the passenger moves to another pick-up point, the vehicle's path is adjusted in time; the passenger behavior prediction includes: based on the dynamic Bayesian network DBN behavior prediction model combined with the passenger's real-time location and historical behavior data, the next location is predicted probabilistically; through model updating, the speed and direction of passenger movement are dynamically adapted; wherein, the prediction of passenger location based on the dynamic Bayesian network DBN is based on Bayesian estimation, and the passenger location prediction formula based on Bayesian estimation is: Among them, X t is the passenger position at time t, X t－1 is the passenger position at time t-1, O t is the sensor observation data at time t, P(X t X t－1 , O t ) represents the posterior probability, P(O t X t ) represents the observation probability, P(X t X t－1 ) represents the state transition probability, P(O t ) represents the marginal probability of observation.

6. The last mile passenger pickup system for an autonomous driving vehicle based on a multimodal large model according to claim 1, characterized in that: The passenger interaction and augmented reality AR guidance module includes: real-time information push and AR navigation guidance; the real-time information push includes: using the WebSocket protocol to achieve low-latency information push through the connection with the autonomous driving app in the passenger's mobile device, ensuring that the passenger can obtain the vehicle location, pick-up point information and path adjustment in real time; at the same time, receiving information feedback from the passenger through the app, and adjusting the path and pick-up point in time according to the information feedback from the passenger; the AR navigation guidance includes: building a 3D model of the passenger's surrounding environment through an augmented reality system based on SLAM technology combined with the acquired passenger environment information, and using the Unity-based AR framework to generate intuitive visual guidance to help passengers find the correct boarding location; through real-time information push and AR navigation, the path and pick-up point of the autonomous driving car are dynamically adjusted, and at the same time, passengers can accurately reach the pick-up point location, thereby improving the pick-up efficiency of the autonomous driving car and increasing the passenger's riding experience.

Images