An adaptive multimodal autonomous vehicle pick-up system for complex environments

Abstract

The present invention proposes an adaptive multimodal autonomous driving vehicle pick-up system for complex environments, which belongs to the field of autonomous driving technology. The system includes: a passive signal perception and fusion module; an active signal input and confirmation module; a VLA multimodal large model fusion module; an embodied intelligence module and a real-time feedback module; an adaptive learning and unsupervised optimization module; a probabilistic planning module; and a remote control module. The autonomous driving technology of this system involves embodied intelligence, a VLA multimodal architecture, adaptive learning, probabilistic planning, and remote control, and on this basis, an end-to-end data processing method is implemented, so that the autonomous driving car can quickly generate corresponding strategies based on the input and respond according to the strategy. The adaptive and control capabilities will be greatly enhanced, making the driving of autonomous driving vehicles safer.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an adaptive multimodal autonomous driving vehicle pick-up system for complex environments. Background Art

[0002] With the rapid development of autonomous driving technology, autonomous driving systems face multiple challenges, including environmental complexity, real-time requirements, and adaptive learning. This paper proposes an autonomous driving pick-up system that combines embodied intelligence, a VLA multimodal architecture, remote control, probabilistic planning, adaptive learning, and a feedback mechanism. This system achieves efficient path planning and task execution in complex environments through physical interaction, data fusion, and feedback learning. Summary of the Invention

[0003] The purpose of the present invention is to overcome the above-mentioned problems existing in the prior art and to greatly improve its technical effects on the basis of the existing technology. The present invention provides an adaptive multi-modal autonomous driving vehicle pick-up system for complex environments, which includes:

[0004] Passive signal perception and fusion module; active signal input and confirmation module; VLA multimodal large model fusion module; embodied intelligence module and real-time feedback module; adaptive learning and unsupervised optimization module; probabilistic planning module; remote control module;

[0005] The passive signal sensing and fusion module is used to capture passive information from the passenger's mobile device and preliminarily locate the passenger's relative position information by processing the passive information. The passive information includes GPS, Wi-Fi, and Bluetooth information from the passenger's mobile device. The processing of the passive information includes initial processing of the passive information, GPS positioning, and correction.

[0006] The active signal input and confirmation module is used to further confirm the precise location of the passenger based on the preliminary positioning of the relative position information of the passenger through the active signal uploaded by the passenger; the active signal includes: images, videos and voice information uploaded by the user; further confirming the precise location of the passenger through the active signal uploaded by the passenger includes: analyzing the active signal data through a deep learning algorithm to determine the precise location of the passenger;

[0007] The VLA multimodal large model fusion module is used to combine the three information modalities of vision, voice, and action received by the system to generate a task plan through multimodal perception input; the vision is the image and video information transmitted by the passenger, the voice is the voice information input by the passenger, and the action is the passenger's current status information;

[0008] The embodied intelligence module and the real-time feedback module are used to adjust the behavior of the autonomous vehicle in real time through interaction between the autonomous vehicle and the environment, and learn based on the received real-time environmental feedback, thereby performing the pick-up task more efficiently and safely; the autonomous vehicle behavior refers to the autonomous vehicle's mission planning;

[0009] The adaptive learning and unsupervised optimization module is used to optimize path planning and task planning using unsupervised learning methods. In the absence of high-quality labeled data, the system can adaptively adjust and learn the optimal path based on the environmental data of the autonomous vehicle, and optimize task planning through reinforcement learning.

[0010] The probabilistic planning module is used to use a probabilistic planning algorithm to evaluate the success probability of each path selection in an uncertain and complex environment, and to select the path with the highest probability as the optimal path;

[0011] The remote control module is used to provide real-time intervention through the real-time data obtained and the feedback mechanism when the success probability assessed by the probabilistic planning module in a complex environment is less than a threshold. The intervention refers to adjusting the path through the remote control personnel control system.

[0012] Specifically, the initial processing of the passive information includes: Kalman filtering, which eliminates errors and noise in the signal data through Kalman filtering. The Kalman filtering is a method based on Bayesian estimation. Kalman filtering updates the state estimate using Bayesian estimation at each moment, thereby eliminating errors and noise in the signal data and obtaining processed GPS, Wi-Fi and Bluetooth information; the GPS positioning and correction includes: preliminarily estimating the passenger's geographic coordinates using the initially processed GPS signal, and correcting the position using the surrounding Wi-Fi and Bluetooth information; the GPS positioning uses the triangulation formula, which is calculated as follows:

[0013]

[0014] Among them, d i is the distance from the passenger device to the i-th base station, (x i ,y i , z i ) represents the coordinates of the i-th base station, (x0, y0, z0) are the coordinates of the passenger device;

[0015] After calculating the distances between multiple base stations and the passenger device, the distance equations between the multiple base stations and the passenger device are combined into a set of three quadratic equations to calculate the coordinates of the passenger device (x0, y0, z0). Usually, the coordinates of at least three base stations are required to calculate the coordinates corresponding to the passenger device through the set of three quadratic equations.

[0016] The method for correcting location using surrounding Wi-Fi and Bluetooth information is as follows: first, using a signal strength weighting algorithm, the location data of multiple signal sources are fused to obtain a fused location; second, the obtained fused location is used to correct the passenger's geographic coordinates initially estimated by GPS. The weighting algorithm formula is:

[0017]

[0018] Among them, P i is the estimated position of different signal sources of the same type, ω i is the weighting coefficient of signal strength, ω i The value range of is: (0, 1). The stronger the signal strength, the larger the corresponding weighting coefficient. P is the weighted position of the signal source. Substitute the weighted positions of different types of signal sources into the triangulation formula to obtain the coordinates of the new passenger device. The average value of the new and original passenger device coordinates is used to correct the passenger device coordinate information.

[0019] Specifically, the analysis of active signal data using a deep learning algorithm includes: first, using a pre-trained deep convolutional neural network (CNN) to process the image and video information in the active signal to identify various landmarks in the passenger's current environment; finding the geographical location information of each landmark through the passenger device location information initially obtained by the GIS geographic information system and the passive signal perception and fusion module; restoring the depth information in the image through a stereo matching algorithm, and then inferring the positional relationship between the photographer and each landmark; finally, based on the positional relationship between the photographer and each landmark, as well as the position coordinates of each landmark, predicting multiple location information C corresponding to the photographer. i , C i Represents the photographer's position information predicted by the i-th marker, by combining multiple C i The precise location of the passenger is finally obtained by adding and averaging. The processing of the image and video information in the active signal using the pre-trained deep convolutional neural network CNN includes: extracting features of the image information through the deep convolutional neural network CNN. The CNN convolution operation formula is:

[0020] Y=f(W*X+b)

[0021] Among them, W represents the convolution kernel, X represents the input image, b represents the bias term, f is the activation function, and Y represents the output of the convolution operation.

[0022] The analysis of active signal data through deep learning algorithms also includes: using the Transformer-based natural language processing model BERT to parse the passenger's voice description, understand the passenger's location instructions, and combine it with the passenger location information obtained through image and video information to further accurately determine the passenger's location information.

[0023] Specifically, the VLA multimodal large model fusion module includes: processing the visual input image video, voice input and action information through the VLA model to obtain a task plan including a target path, control instructions and task execution strategy; the steps are: first, performing data fusion analysis and processing on the visual input image video, voice input and action information through the VLA multimodal large model to analyze passenger needs and the riding environment; second, combining the active signal input and the passenger position information obtained by the confirmation module to preliminarily generate a target path, and generate the corresponding control instructions and task execution strategy for the vehicle based on the generated target path.

[0024] Specifically, the embodied intelligence module and real-time feedback module include: real-time perception and decision-making, environmental feedback mechanism, and reinforcement learning and self-supervised learning. The real-time perception and decision-making includes: obtaining real-time information about the autonomous vehicle's surrounding environment through the perception system, and making timely decisions based on the obtained information. The perception system integrates multiple sensors and obtains surrounding environment data through sensors. By fusing and processing sensor data, an environmental model is established. The established model formula is:

[0025] E t =f p (S t ,θ p )

[0026] Among them, E t is the environmental state at time t, including traffic, obstacles, pedestrians and lane markings; f p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, θ p are the parameters of the perception model.

[0027] The environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the task plan and then adjusting the path plan based on the feedback; first, the system generates corresponding decisions through the environmental model established by the real-time perception and decision-making module, and interacts with the targets in the corresponding environment; then, after interacting and receiving feedback from the targets in the environment, the system adjusts the task plan based on the feedback results and determines the next decision of the car; finally, the path plan is adjusted based on the readjusted task plan.

[0028] The reinforcement learning and self-supervised learning include: through reinforcement learning and self-supervised learning, the system continuously optimizes the task strategy during the task execution process to improve the long-term performance of the system; the system optimizes the path through the reinforcement learning framework, and the system optimizes the action value in the current state through the action value in the previous state; through continuous updates, the system optimizes the path selection and improves the efficiency of task execution.

[0029] Specifically, the adaptive learning and unsupervised optimization module includes: in addition to reinforcement learning and self-supervised learning to optimize task strategies, the autonomous vehicle uses adaptive learning and unsupervised learning methods to optimize path planning and task execution during driving, so as to minimize the unsupervised learning loss function, enabling the system to adaptively learn from the environment and optimize path planning. At the same time, reinforcement learning is used to optimize task planning based on the optimized path. The formula for minimizing the loss function is:

[0030] η(E t )=||E t -E t ′|| 2 +λ||θ model || 2

[0031] Among them E t is the current state of the environment, E t ' is the environmental state predicted by the system, λ is the regularization coefficient, θ model are the parameters of the learning model, || || is the norm, and η() is the function that minimizes the loss; unsupervised learning adaptively learns from the environment and optimizes path planning in this way.

[0032] Specifically, the probability planning module includes: formula modeling through Bayesian reasoning, and the modeling formula is:

[0033]

[0034] Among them, P s (x t , a t ) is the success probability of path planning, P(x t |a t ) is in action a t Under the condition of state x t The probability of occurrence, P(a t ) is the selection action a t The prior probability, P(x t ) is the marginal probability of the current state; the success probability P corresponding to each path planning s (x t , a t) is compared with the set threshold, and the P that exceeds the set threshold and has the highest success probability of path planning is selected s (x t , a t ) is used as the optimal path; the threshold is a fixed value set according to the parameters of the system and Bayesian modeling formula.

[0035] Specifically, the VLA multimodal large model fusion module, embodied intelligence module and real-time feedback module, adaptive learning and unsupervised optimization module and probabilistic planning module are designed as an end-to-end autonomous driving module, that is, after the car surrounding data and passenger information data collected by the sensor are input into the autonomous driving module, the autonomous driving module directly generates a series of driving decisions, which include: mission planning, path planning, control instructions and path optimization

[0036] Specifically, the remote control module includes: in a complex environment, when the success probability evaluated by the probability planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk and requires human intervention; when the success probability evaluated by the probability planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk; when the success probability evaluated by a certain probability planning module is greater than the threshold, it is determined that the autonomous driving vehicle in the corresponding environment is not high-risk. At this time, the adaptive control of multiple other modules of the system can ensure the safety of the autonomous driving vehicle.

[0037] The beneficial effects of the present invention are:

[0038] The present invention provides an adaptive multi-modal autonomous driving vehicle pick-up system for complex environments, which has the following advantages:

[0039] 1. When determining the passenger's location information, the present invention first obtains the passenger's location information preliminarily by processing the passive information, and then accurately obtains the passenger's location information through the active information sent by the passenger based on the preliminarily obtained passenger location information. Compared with processing all the information together, this method will obtain more accurate location information.

[0040] 2. The present invention provides an autonomous driving pick-up system that combines embodied intelligence, VLA multimodal architecture, adaptive learning, probabilistic planning, and remote control. The adaptive and control capabilities will be greatly enhanced, making the driving of autonomous vehicles safer.

[0041] 3. The self-supervised and unsupervised methods are used to optimize the task planning and path planning of autonomous driving vehicles, thereby improving the adaptability and safety of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of an adaptive multimodal autonomous driving vehicle pick-up system for complex environments according to the present invention. DETAILED DESCRIPTION

[0043] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention and cannot be used to limit the present invention.

[0044] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0045] like Figure 1 As shown, a schematic diagram of an adaptive multimodal autonomous driving vehicle pick-up system for complex environments according to an embodiment of the present invention is shown; the schematic diagram includes: a passive signal perception and fusion module S1; an active signal input and confirmation module S2; a VLA multimodal large model fusion module S3; an embodied intelligence module and real-time feedback module S4; an adaptive learning and unsupervised optimization module S5; a probabilistic planning module S6; and a remote control module S7.

[0046] Among them, the passive signal perception and fusion module S1 is used to capture passive information from the passenger's mobile device and preliminarily locate the passenger's relative position information by processing the passive information; the passive information includes: GPS, Wi-Fi and Bluetooth information of the passenger's mobile device; the processing of the passive information includes: initial processing of passive information, GPS positioning and correction; specifically, the initial processing of the passive information includes: Kalman filtering, which eliminates errors and noise in the signal data through Kalman filtering. The Kalman filtering is a method based on Bayesian estimation. The Kalman filtering updates the state estimation by using Bayesian estimation at each moment, thereby eliminating errors and noise in the signal data and obtaining processed GPS, Wi-Fi and Bluetooth information; the GPS positioning and correction includes: preliminarily estimating the passenger's geographic coordinates through the preliminarily processed GPS signal, and correcting the position through the surrounding Wi-Fi and Bluetooth information; the GPS positioning uses the triangulation formula, and the calculation formula of the triangulation formula is:

[0047]

[0048] Among them, d i is the distance from the passenger device to the i-th base station, (x i ,y i , z i ) represents the coordinates of the i-th base station, (x0, y0, z0) are the coordinates of the passenger device;

[0049] After calculating the distances between multiple base stations and the passenger device, the distance equations between the multiple base stations and the passenger device are combined into a set of three quadratic equations to calculate the coordinates of the passenger device (x0, y0, z0). Usually, the coordinates of at least three base stations are required to calculate the coordinates corresponding to the passenger device through the set of three quadratic equations.

[0050] The method for correcting location using surrounding Wi-Fi and Bluetooth information is as follows: first, using a signal strength weighting algorithm, the location data of multiple signal sources are fused to obtain a fused location; second, the obtained fused location is used to correct the passenger's geographic coordinates initially estimated by GPS. The weighting algorithm formula is:

[0051]

[0052] Among them, P i is the estimated position of different signal sources of the same type, ω i is the weighting coefficient of signal strength, ω i The value range of is: (0, 1). The stronger the signal strength, the larger the corresponding weighting coefficient. P is the weighted position of the signal source. Substitute the weighted positions of different types of signal sources into the triangulation formula to obtain the coordinates of the new passenger device. The average value of the new and original passenger device coordinates is used to correct the passenger device coordinate information.

[0053] Among them, the active signal input and confirmation module S2 is used to further confirm the precise position of the passenger through the active signal uploaded by the passenger on the basis of the preliminary positioning of the relative position information of the passenger; the active signal includes: images, videos and voice information uploaded by the user; the further confirmation of the precise position of the passenger through the active signal uploaded by the passenger includes: analyzing the active signal data through a deep learning algorithm to determine the precise position of the passenger; specifically, the analysis of the active signal data through the deep learning algorithm includes: first, using a pre-trained deep convolutional neural network CNN to process the image and video information in the active signal to identify various landmarks in the passenger's current environment; searching for the geographical location information of each landmark through the passenger device location information preliminarily obtained by the GIS geographic information system and the passive signal perception and fusion module; restoring the depth information in the image through a stereo matching algorithm, and then inferring the positional relationship between the photographer and each landmark; finally, predicting multiple position information C corresponding to the photographer based on the positional relationship between the photographer and each landmark, as well as the position coordinates of each landmark. i , C i Represents the photographer's position information predicted by the i-th marker, by combining multiple C iThe precise location of the passenger is finally obtained by adding and averaging. The processing of the image and video information in the active signal using the pre-trained deep convolutional neural network CNN includes: extracting features of the image information through the deep convolutional neural network CNN. The CNN convolution operation formula is:

[0054] Y=f(W*X+b)

[0055] Among them, W represents the convolution kernel, X represents the input image, b represents the bias term, f is the activation function, and Y represents the output of the convolution operation.

[0056] Furthermore, analyzing active signal data through deep learning algorithms also includes: using the Transformer-based natural language processing model BERT to parse passengers' voice descriptions, understand passengers' location instructions, and combine them with passenger location information obtained through image and video information to further accurately determine passengers' location information.

[0057] Among them, the VLA multimodal large model fusion module S3 is used to combine the three information modalities of vision, voice and action received by the system, and generate a task plan through multimodal perception input; the vision is the image and video information transmitted by the passenger, the voice is the voice information input by the passenger, and the action is the passenger's current status information; specifically, the image and video input of the visual input, the voice input and the action information are processed by the VLA model to obtain a task plan including a target path, control instructions and task execution strategy; the steps are: first, the image and video input of the visual input, the voice input and the action information are processed by the VLA multimodal large model to analyze the passenger's needs and the riding environment; secondly, the active signal input is combined with the passenger position information obtained by the confirmation module to preliminarily generate the target path, and the control instructions and task execution strategy corresponding to the vehicle are generated according to the generated target path.

[0058] Among them, the embodied intelligence module and real-time feedback module S4 are used to adjust the behavior of the autonomous driving vehicle in real time through the interaction between the autonomous driving vehicle and the environment, and learn based on the received real-time environmental feedback, so as to more efficiently perform the pick-up task; the behavior of the autonomous driving vehicle refers to the task planning of the autonomous driving vehicle; specifically, the embodied intelligence module and real-time feedback module include: real-time perception and decision-making, environmental feedback mechanism, and reinforcement learning and self-supervised learning; the real-time perception and decision-making include: obtaining the surrounding environment information of the autonomous driving vehicle in real time through the perception system, and making timely decisions based on the obtained information; the perception system integrates multiple sensors and obtains the surrounding environment data through the sensors. By fusing and processing the sensor data, an environmental model is established. The established model formula is:

[0059] E t =fp (S t ,θ p )

[0060] Among them, E t is the environmental state at time t, including traffic, obstacles, pedestrians and lane markings; f p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, θ p are the parameters of the perception model.

[0061] In the above embodiment, specifically, the environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the task plan according to the feedback, and then adjusting the path planning; first, the system generates corresponding decisions through the environmental model established by the real-time perception and decision-making module, and interacts with the targets in the corresponding environment; then, after the interaction, after receiving feedback from the targets in the environment, the system adjusts the task plan according to the feedback results and determines the next decision of the car; finally, the path planning is adjusted according to the readjusted task plan.

[0062] In the above embodiment, specifically, reinforcement learning and self-supervised learning include: through reinforcement learning and self-supervised learning, the system continuously optimizes the task strategy in the process of executing the task, thereby improving the long-term performance of the system; the system optimizes the path through the reinforcement learning framework, and the system optimizes the action value at the current moment through the action value at the previous moment; through continuous updates, the system performs path optimization selection and improves the efficiency of task execution.

[0063] Among them, the adaptive learning and unsupervised optimization module S5 is used to optimize path planning and task planning using unsupervised learning methods. In the absence of high-quality labeled data, the system can adaptively adjust and learn the optimal path through the environmental data of the autonomous driving vehicle, and optimize task planning through reinforcement learning. Specifically, the adaptive learning and unsupervised optimization module includes: in addition to reinforcement learning and self-supervised learning to optimize task strategies, the autonomous driving vehicle uses adaptive learning and unsupervised learning methods to optimize path planning and task execution during driving, so as to minimize the unsupervised learning loss function, so that the system can adaptively learn from the environment and optimize path planning, and at the same time optimize task planning through reinforcement learning based on the optimized path. The formula for minimizing the loss function is:

[0064] η(E t )=||E t -E t ′|| 2 +λ||θ model || 2

[0065] Among them E t is the current state of the environment, E t ' is the environmental state predicted by the system, λ is the regularization coefficient,

[0066] θ model are the parameters of the learning model, |||| is the norm, and η() is the function that minimizes the loss; unsupervised learning adaptively learns from the environment and optimizes path planning in this way.

[0067] Among them, the probability planning module S6 is used to use the probability planning algorithm to evaluate the success probability of each path selection in an uncertain and complex environment, and select the path with the highest probability as the optimal path; specifically, the formula modeling is performed through Bayesian reasoning, and the modeling formula is:

[0068]

[0069] Among them, P s (x t , a t ) is the success probability of path planning, P(x t |a t ) is in action a t Under the condition of state x t The probability of occurrence, P(a t ) is the selection action a t The prior probability, P(x t ) is the marginal probability of the current state; the success probability P corresponding to each path planning s (x t , a t ) is compared with the set threshold, and the P that exceeds the set threshold and has the highest success probability of path planning is selected s (x t , a t ) is used as the optimal path; the threshold is a fixed value set according to the parameters of the system and Bayesian modeling formula.

[0070] In the above embodiment, specifically, the VLA multimodal large model fusion module S3, the embodied intelligence module and real-time feedback module S4, the adaptive learning and unsupervised optimization module S5 and the probabilistic planning module S6 are designed as an end-to-end autonomous driving module, so that after the car surrounding data and passenger information data collected by the sensor are input into the autonomous driving module, the autonomous driving module directly generates a series of driving decisions; the autonomous driving module refers to a series of modules that can realize end-to-end decision-making of autonomous driving vehicles; the series of driving decisions include: mission planning, path planning, control instructions and path optimization.

[0071] In the above embodiment, the above end-to-end autonomous driving module is constructed by using the OpenPilot open source autonomous driving technology. Through the data collection of sensors and the output of the module, the decision to control the autonomous driving vehicle is directly generated, making the decision faster and more accurate.

[0072] Among them, the remote control module S7 is used to provide real-time intervention through the real-time data and feedback mechanism obtained when the success probability evaluated by the probabilistic planning module in a complex environment is less than the threshold. The intervention refers to adjusting the path through the remote control personnel control system; specifically, in a complex environment, when the success probability evaluated by the probabilistic planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk and requires human intervention; when the success probability evaluated by the probabilistic planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk; when the success probability evaluated by a certain probabilistic planning module is greater than the threshold, it is determined that the autonomous driving vehicle in the corresponding environment is not high-risk. At this time, the adaptive regulation of multiple other modules of the system can ensure the safety of the autonomous driving vehicle.

Claims

1. An adaptive multi-modal autonomous driving vehicle pick-up system for complex environments, characterized by: The system comprises: Passive signal perception and fusion module; active signal input and confirmation module; VLA multimodal large model fusion module; embodied intelligence module and real-time feedback module; adaptive learning and unsupervised optimization module; probabilistic planning module; remote control module; The passive signal sensing and fusion module is used to capture passive information from the passenger's mobile device and preliminarily locate the passenger's relative position information by processing the passive information; the passive information includes GPS, Wi-Fi, and Bluetooth information from the passenger's mobile device; the processing of the passive information includes initial processing of the passive information, GPS positioning, and correction; The active signal input and confirmation module is used to further confirm the precise location of the passenger based on the preliminary positioning of the relative position information of the passenger through the active signal uploaded by the passenger; the active signal includes: images, videos and voice information uploaded by the user; further confirming the precise location of the passenger through the active signal uploaded by the passenger includes: analyzing the active signal data through a deep learning algorithm to determine the precise location of the passenger; The VLA multimodal large model fusion module is used to combine the three information modalities of vision, voice, and action received by the system to generate a task plan through multimodal perception input; the vision is the image and video information transmitted by the passenger, the voice is the voice information input by the passenger, and the action is the passenger's current status information; The embodied intelligence module and the real-time feedback module are used to adjust the behavior of the autonomous vehicle in real time through interaction between the autonomous vehicle and the environment, and learn based on the received real-time environmental feedback, thereby performing the pick-up task more efficiently and safely; the autonomous vehicle behavior refers to the autonomous vehicle's mission planning; The adaptive learning and unsupervised optimization module is used to optimize path planning and task planning using unsupervised learning methods. In the absence of high-quality labeled data, the system can adaptively adjust and learn the optimal path based on the environmental data of the autonomous vehicle, and optimize task planning through reinforcement learning. The probabilistic planning module is used to use a probabilistic planning algorithm to evaluate the success probability of each path selection in an uncertain and complex environment, and to select the path with the highest probability as the optimal path; The remote control module is used to provide real-time intervention through the real-time data obtained and the feedback mechanism when the success probability assessed by the probabilistic planning module in a complex environment is less than a threshold. The intervention refers to adjusting the path through the remote control personnel control system.

2. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The initial processing of the passive information includes: Kalman filtering, which eliminates errors and noise in the signal data through Kalman filtering. The Kalman filtering is a method based on Bayesian estimation. Kalman filtering updates the state estimate using Bayesian estimation at each moment, thereby eliminating errors and noise in the signal data and obtaining processed GPS, Wi-Fi and Bluetooth information. The GPS positioning and correction includes: preliminarily estimating the geographic coordinates of the passenger using the initially processed GPS signal, and correcting the position using the surrounding Wi-Fi and Bluetooth information. The GPS positioning uses the triangulation formula, which is calculated as follows: Among them, d i is the distance from the passenger device to the i-th base station, (x i ,y i , z i ) represents the coordinates of the i-th base station, (x0, y0, z0) are the coordinates of the passenger device; After calculating the distances between multiple base stations and the passenger device, the distance equations between the multiple base stations and the passenger device are combined into a set of three quadratic equations to calculate the coordinates of the passenger device (x0, y0, z0). Usually, the coordinates of at least three base stations are required to calculate the coordinates corresponding to the passenger device through the set of three quadratic equations. The method for correcting location using surrounding Wi-Fi and Bluetooth information is as follows: first, using a signal strength weighting algorithm, the location data of multiple signal sources are fused to obtain a fused location; second, the obtained fused location is used to correct the passenger's geographic coordinates initially estimated by GPS. The weighting algorithm formula is: Among them, P i is the estimated position of different signal sources of the same type, ω i is the weighting coefficient of signal strength, ω i The value range of is: (0, 1). The stronger the signal strength, the larger the corresponding weighting coefficient. P is the weighted position of the signal source. Substitute the weighted positions of different types of signal sources into the triangulation formula to obtain the coordinates of the new passenger device. The average value of the new and original passenger device coordinates is used to correct the passenger device coordinate information.

3. The adaptive multimodal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The analysis of active signal data using a deep learning algorithm includes: first, using a pre-trained deep convolutional neural network (CNN) to process the image and video information in the active signal to identify various landmarks in the passenger's current environment; finding the geographical location information of each landmark using the passenger device location information initially obtained through the GIS geographic information system and the passive signal perception and fusion module; restoring the depth information in the image using a stereo matching algorithm, and then inferring the positional relationship between the photographer and each landmark; finally, predicting multiple pieces of position information C corresponding to the photographer based on the positional relationship between the photographer and each landmark, as well as the position coordinates of each landmark. i , C i Represents the photographer's position information predicted by the i-th marker, by combining multiple C i The precise location of the passenger is finally obtained by adding and averaging. The processing of the image and video information in the active signal using the pre-trained deep convolutional neural network CNN includes: extracting features of the image information through the deep convolutional neural network CNN. The CNN convolution operation formula is: Y=f(W*X+b) Where W represents the convolution kernel, X represents the input image, b represents the bias term, f is the activation function, and Y represents the output of the convolution operation; The analysis of active signal data through deep learning algorithms also includes: using the Transformer-based natural language processing model BERT to parse the passenger's voice description, understand the passenger's location instructions, and combine it with the passenger location information obtained through image and video information to further accurately determine the passenger's location information.

4. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The VLA multimodal large model fusion module includes: processing visual input image video, voice input and action information through the VLA model to obtain a task plan including a target path, control instructions and task execution strategy; the steps are: first, using the VLA multimodal large model to perform data fusion analysis and processing on the visual input image video, voice input and action information to analyze passenger needs and the riding environment; second, combining the active signal input with the passenger position information obtained by the confirmation module to preliminarily generate a target path, and generating control instructions and task execution strategies corresponding to the vehicle based on the generated target path.

5. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The embodied intelligence module and real-time feedback module include: real-time perception and decision-making, environmental feedback mechanism, and reinforcement learning and self-supervised learning. The real-time perception and decision-making includes: obtaining real-time information about the autonomous vehicle's surrounding environment through the perception system and making timely decisions based on the obtained information. The perception system integrates multiple sensors and obtains surrounding environment data through sensors. By fusing and processing sensor data, an environmental model is established. The established model formula is: E t =f p (S t ,the p ) Among them, E t is the environmental state at time t, including traffic, obstacles, pedestrians and lane markings; f p is the environment perception function, which indicates how to convert the sensor data into the environment state; S t is the data collected by the sensor at time t, θ p are the parameters of the perception model; The environmental feedback mechanism includes: obtaining feedback through interaction with the environment, and adjusting the mission plan and path planning based on the feedback. First, the system generates corresponding decisions based on the environmental model established by the real-time perception and decision-making module, and interacts with the corresponding targets in the environment. Then, after interacting with the targets in the environment, the system adjusts the mission plan based on the feedback results and determines the next decision of the vehicle. Finally, the path planning is adjusted based on the readjusted mission plan. The reinforcement learning and self-supervised learning include: through reinforcement learning and self-supervised learning, the system continuously optimizes the task strategy during the task execution process to improve the long-term performance of the system; the system optimizes the path through the reinforcement learning framework, and the system optimizes the action value in the current state through the action value in the previous state; through continuous updates, the system optimizes the path selection and improves the efficiency of task execution.

6. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The adaptive learning and unsupervised optimization module includes: in addition to reinforcement learning and self-supervised learning to optimize task strategies, the autonomous vehicle uses adaptive learning and unsupervised learning methods to optimize path planning and task execution during driving, so as to minimize the unsupervised learning loss function, enabling the system to adaptively learn from the environment and optimize path planning. At the same time, reinforcement learning is used to optimize task planning based on the optimized path. The formula for minimizing the loss function is: Among them E t is the current state of the environment, E t ' is the environmental state predicted by the system, λ is the regularization coefficient, θ model are the parameters of the learning model, || || is the norm, and η( ) is the function that minimizes the loss; unsupervised learning adaptively learns from the environment and optimizes path planning in this way.

7. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The probability planning module includes: formula modeling through Bayesian reasoning, and the modeling formula is: Among them, P s (x t , a t ) is the success probability of path planning, P(x t |a t ) is in action a t Under the condition of state x t The probability of occurrence, P(a t ) is the selection action a t The prior probability, P(x t ) is the marginal probability of the current state; the success probability P corresponding to each path planning s (x t , a t ) is compared with the set threshold, and the P that exceeds the set threshold and has the highest success probability of path planning is selected s (x t , a t ) is used as the optimal path; the threshold is a fixed value set according to the parameters of the system and Bayesian modeling formula.

8. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The VLA multimodal large model fusion module, embodied intelligence module and real-time feedback module, adaptive learning and unsupervised optimization module and probabilistic planning module are designed as an end-to-end autonomous driving module. That is, after the car surrounding data and passenger information data collected by sensors are input into the autonomous driving module, the autonomous driving module directly generates a series of driving decisions, which include: mission planning, path planning, control instructions and path optimization.

9. The adaptive multi-modal autonomous driving vehicle pick-up system for complex environments according to claim 1, characterized in that: The remote control module includes: in a complex environment, when the success probability evaluated by the probability planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk and requires human intervention; when the success probability evaluated by the probability planning module is less than the threshold, it is determined that the autonomous driving in the corresponding environment is high-risk; when the success probability evaluated by a certain probability planning module is greater than the threshold, it is determined that the autonomous driving vehicle in the corresponding environment is not high-risk. At this time, the safety of the autonomous driving vehicle can be ensured through adaptive control of multiple other modules of the system.

Images