Adaptive pick-up and drop-off system for autonomous vehicles based on enhanced intelligent interaction

Abstract

The present invention discloses an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction, including a passenger intelligent interaction control module, a multimodal large model optimization module, a real-time path planning and dynamic adjustment module, a passenger feedback and personalized recommendation module, and an intelligent accident response and safety module; the present invention allows passengers to independently control pick-up and drop-off points through mobile devices, adjust routes in real time, and with the support of a multimodal large model, intelligently optimize pick-up and drop-off plans based on traffic conditions, road changes, and emergencies. The present invention is particularly suitable for scenarios where passengers have flexible demands on pick-up and drop-off points, and can improve pick-up and drop-off efficiency and passenger experience in a dynamically changing environment.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction. Background Art

[0002] As a strategic development direction for the global automotive industry, autonomous driving technology has become a key technology for resolving traffic problems. With the development and maturity of new-generation digital technologies such as artificial intelligence, the Internet of Things, and 5G mobile communications, the level of automotive automation and intelligence continues to rise, and market acceptance of autonomous driving is also increasing. The development level of autonomous driving technology is directly related to the international competitiveness of each country's automotive industry and the global industrial division of labor. Therefore, major countries around the world attach great importance to the development of autonomous driving and are actively exploring traffic laws, regulations, and policies.

[0003] The convergence of augmented intelligence (IA) and artificial intelligence (AI) is becoming increasingly evident. IA focuses on the auxiliary role of AI, emphasizing that cognitive technology aims to enhance human intelligence and capabilities, while AI aims to create systems that operate without human intervention. Currently, most cognitive intelligence products and project solutions combine these two technologies. As the boundaries between the two become increasingly blurred, AI and IA are also converging, ushering in a new era of IA and human-machine collaboration.

[0004] Current autonomous vehicle pick-up and drop-off systems typically rely on pre-set pick-up points or fixed routes, which are unable to adequately adapt to dynamic environmental changes such as traffic congestion, road closures, or emergencies. For passengers, fixed pick-up points and inflexible routes often result in a poor pick-up and drop-off experience. Summary of the Invention

[0005] The purpose of the present invention is to provide an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction to solve the above-mentioned problems existing in the prior art.

[0006] The specific application is as follows:

[0007] An adaptive pick-up and drop-off system for autonomous vehicles based on enhanced intelligent interaction, including a passenger intelligent interaction control module, a multimodal large model optimization module, a real-time path planning and dynamic adjustment module, a passenger feedback and personalized recommendation module, and an intelligent accident response and safety module;

[0008] The passenger intelligent interactive control module is used to provide a real-time interactive interface between passengers and vehicles, allowing passengers to independently select, adjust or change pick-up points through their mobile devices and control the vehicle's driving path in real time;

[0009] The multimodal large model optimization module analyzes the behavior pattern of each passenger in the group through the multimodal large model, combines the current environmental data and historical behavior data, and predicts the future movement path and time schedule of each passenger in the group;

[0010] The real-time path planning and dynamic adjustment module is used to generate the real-time driving path of the vehicle and allow passengers to dynamically adjust the path during driving;

[0011] The passenger feedback and personalized recommendation module generates personalized pick-up point recommendations and optimization plans based on passengers' historical behavior, preferences, and feedback;

[0012] The intelligent accident response and safety module is used to provide intelligent response measures and safety assurance for emergencies;

[0013] The passenger intelligent interaction control module, the multimodal large model optimization module, the real-time path planning and dynamic adjustment module, the passenger feedback and personalized recommendation module and the intelligent accident response and safety module are interconnected.

[0014] Furthermore, the passenger intelligent interaction control module includes a pick-up point selection submodule, a route adjustment submodule, and a preference setting submodule;

[0015] The pick-up point selection submodule is used for passengers to select from a plurality of predefined pick-up points or manually input a new pick-up point;

[0016] The route adjustment submodule is used for passengers to view the vehicle's route in real time and actively adjust the route in the event of road congestion or accidents;

[0017] The preference setting submodule is used to set the passenger's personalized pick-up and drop-off preferences through a decision tree model and generate recommendations, wherein the personalized pick-up and drop-off preferences at least include selecting a quieter pick-up and drop-off point, being close to a building entrance, and avoiding crowded areas;

[0018] The specific implementation process of the pick-up point selection submodule is as follows:

[0019] Use a deep learning model to parse the passenger's voice or text input to find the pick-up point. The specific process is as follows:

[0020] Voice input, using a Wave2Vec-based speech recognition model to convert speech into text;

[0021] Text parsing: Using the BERT model for semantic analysis to extract the location, time, and route requirements for riding;

[0022] The specific implementation process of the path adjustment submodule is as follows:

[0023] Route planning is performed based on the graph search A* algorithm. If the passenger changes the pick-up point or route, the optimal route is dynamically recalculated. The formula is:

[0024] f(n)=g(n)+h(n),

[0025] Among them, f(n) represents the total cost, g(n) represents the actual cost from the starting point to the current node, and h(n) represents the estimated cost from the current node to the target;

[0026] The specific implementation process of the preference setting submodule is as follows:

[0027] Feature fusion: Inputting passengers' immediate needs and long-term preferences as features into a decision tree model and training the model, wherein the immediate needs are current specific conditions, including at least weather, time, and location, and the long-term preferences represent passenger preference statistics based on historical data;

[0028] Weight setting: set different weights for different features, and set the weight of long-term preferences higher than the weight of immediate needs;

[0029] Dynamic adjustment: Dynamically adjust weight settings based on real-time feedback during decision tree model training;

[0030] Model evaluation and iteration: regularly use new data evaluation models to evaluate the decision tree model, and update the parameters of the decision tree model based on the evaluation results;

[0031] The model is deployed and used to prioritize passengers' different preferences using the trained decision tree model, and personalized pick-up and drop-off recommendations are made based on passenger historical records.

[0032] Furthermore, the multimodal large model optimization module includes a multimodal input processing submodule, a traffic and environment perception submodule, and an intelligent optimization engine submodule;

[0033] The multimodal input processing submodule is used to analyze the passenger's voice or text input through natural language processing technology to generate text input data; confirm the environmental image uploaded by the passenger through image analysis technology to generate image input data; and perform data fusion processing through the conditional variational autoencoding (CVAE) model to generate multimodal fusion data;

[0034] The traffic and environment perception submodule is used to obtain real-time traffic and environment data from traffic sensors, map APIs, and weather APIs. The real-time traffic and environment data is used to optimize the vehicle's driving path and pick-up points;

[0035] The intelligent optimization engine submodule is used to perform predictive analysis on the real-time traffic and environmental data through a multimodal large model, and to intelligently adjust the passenger pick-up and drop-off plan.

[0036] Furthermore, the specific implementation process of data fusion processing through the conditional variational autoencoding (CVAE) model in the multimodal input processing submodule is as follows:

[0037] Speech processing, using Wave2Vec to convert passenger speech into text data;

[0038] Image processing: For environmental images uploaded by passengers, DeepLabv3+ image semantic segmentation network is used to extract features, and the attention mechanism is integrated to enhance object recognition in environmental images to generate image semantic segmentation data;

[0039] Text parsing: Using the GPT-4 model to perform semantic analysis on passenger text input and generate text semantic parsing data;

[0040] The conditional variational autoencoding (CVAE) model embeds the text data, the image semantic segmentation data, and the text semantic parsing data into the same vector space, and realizes multimodal fusion by maximizing the similarity between related modalities.

[0041] The specific implementation process of the intelligent optimization engine submodule for predicting and analyzing the real-time traffic and environmental data through a multimodal large model and intelligently adjusting the passenger pick-up and drop-off plan is as follows:

[0042] The multimodal large model includes a time encoding model, a space encoding model, two conditional variational autoencoding (CVAE) models, a fusion decoding model and a semantic BEV modal prediction model. The time encoding model takes the historical behavior data of the target passenger to be predicted as input and generates time encoding. The space encoding model takes the multimodal fusion data as input and generates space encoding. The time encoding model and the space encoding model will generate time-space features. The time encoding model adopts a bidirectional long short-term memory (LSTM) network, and the space encoding model adopts a DenseNet network. The two conditional variational autoencoding (CVAE) models include a first CVAE model and a second CVAE model. The output of the time encoding model is connected to the input of the first CVAE model. The output of the space encoding model is connected to the input of the second CVAE model. The fusion decoding model converts the connected time-space features into a prediction trajectory with confidence. The semantic BEV modal prediction model is used to optimize the confidence of the prediction trajectory.

[0043] The real-time prediction process is:

[0044] Decomposing the entire prediction task into two sub-models: a modal prediction model TPM based on modal distribution and a trajectory prediction model MPM based on modal trajectory distribution, both of which are based on the multimodal large model;

[0045] Given historical passenger behavior data and real-time traffic and environmental data, TPM performs probabilistic inference on predicted trajectories for all modalities;

[0046] The trajectory prediction model (MPM) of the modal trajectory distribution is used to solve the comprehensive probability of different modes and compare it with a preset probability threshold to determine whether the comprehensive probability is greater than the preset probability threshold. If so, the passenger pick-up and drop-off plan is adjusted. At the same time, the semantic BEV modal prediction model is used to optimize the confidence level of each modal trajectory prediction.

[0047] Furthermore, the real-time path planning and dynamic adjustment module includes a real-time path planning submodule and a dynamic adjustment and optimization submodule;

[0048] The real-time route planning submodule generates an optimal driving route based on the passenger's currently selected pick-up point and real-time traffic conditions, and recalculates the route for each change, including changes to the pick-up point and real-time traffic conditions;

[0049] The dynamic adjustment and optimization submodule is used to generate an alternative route when an emergency occurs to the vehicle, and adjust the alternative route according to the feedback from the passengers to generate a final driving route.

[0050] Furthermore, the specific implementation process of the real-time path planning submodule is as follows:

[0051] Combine multi-sensor data and use SLAM technology for environmental mapping and self-positioning to achieve dynamic path adjustment;

[0052] Based on the passenger's selected pick-up point and real-time traffic data, the A* algorithm is used to generate the optimal path. The path planning formula uses the heuristic function:

[0053]

[0054] Where H(n) represents the estimated cost from the current node n to the target node, (x1, y1) represents the coordinates of the current node, and (x2, y2) represents the coordinates of the target node;

[0055] The specific process of the dynamic adjustment and optimization submodule is as follows:

[0056] When an emergency event is detected, a new path plan is generated using a genetic algorithm. The genetic algorithm generates a path population through crossover, mutation, and selection operations, scores the adaptability of each path, and selects the optimal path.

[0057] The alternative path generation formula is:

[0058] C(x,y)=min[C(x-1,y),C(x,y-1)]+w(x,y),

[0059] Among them, C(x,y) represents the minimum cost from the starting point to the current node (x,y), and w(x,y) represents the weight of the current node.

[0060] Furthermore, the passenger feedback and personalized recommendation module includes a personalized recommendation submodule and a real-time feedback submodule;

[0061] The personalized recommendation submodule is used to prioritize the passenger's preferred pick-up points and routes based on the passenger's historical data, wherein the historical data includes previously used pick-up points and previously preferred routes;

[0062] The real-time feedback submodule is used to provide passengers with the vehicle's real-time location, estimated arrival time, and route suggestions via their smartphones, and to adjust the pick-up and drop-off plan in real time based on passenger feedback;

[0063] The specific implementation process of the personalized recommendation submodule is as follows:

[0064] Collaborative filtering technology is used to analyze passengers' historical pick-up and drop-off data, and personalized recommendations are achieved through a matrix decomposition model. The formula is:

[0065] R=P*Q T ,

[0066] Among them, R represents the rating matrix of the pick-up point, P and Q are the feature matrices of passengers and pick-up points respectively. By optimizing P and Q, we can predict the passenger's rating of the recommended pick-up point.

[0067] Furthermore, the intelligent accident response and safety module includes an accident response submodule and a safety priority mechanism submodule;

[0068] The accident response submodule is used to automatically generate an alternative route and notify passengers to adjust their routes when a road accident or road closure is detected;

[0069] The safety priority mechanism submodule prioritizes the safest pick-up and drop-off plan based on real-time traffic and environmental data. Furthermore, the specific implementation process of the accident response submodule is as follows:

[0070] If a road accident or road closure is detected, the Bayesian network is used to calculate the probabilities of different response plans and select the optimal alternative path. The formula is:

[0071]

[0072] Where P(H|E) represents the probability that hypothesis H holds given that event E has occurred, P(E|H) represents the probability that event E will occur under hypothesis H, and P(H) and P(E) represent the prior probabilities of hypothesis H and time E.

[0073] The specific implementation process of the safety priority mechanism submodule is as follows:

[0074] The Markov decision process (MDP) is used to calculate the risk of each state based on environmental data and select the solution with the highest security. The Markov decision formula is:

[0075] V(s)=max a [R(s,a)+b∑ s' P(s'|s,a)V(s'))],

[0076] Where V(s) represents the value of state s, R(s,a) represents the immediate reward of executing action a in state s, b represents the discount factor, and P(s'|s,a) represents the transition probability from state s to state s'.

[0077] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0078] Embodiments of the present invention provide a passenger intelligent interaction control module for providing a real-time interactive interface between passengers and vehicles. Passengers can independently select, adjust, or change pick-up and drop-off points through their mobile devices and control the vehicle's travel path in real time. A multimodal large model optimization module analyzes the behavioral patterns of each passenger in a group using a multimodal large model and, combining current environmental data with historical behavioral data, predicts each passenger's future travel path and schedule. A real-time route planning and dynamic adjustment module generates a real-time vehicle route and allows passengers to dynamically adjust the route during travel. A passenger feedback and personalized recommendation module generates personalized pick-up and drop-off point recommendations and optimization plans based on passengers' historical behavior, preferences, and feedback. An intelligent accident response and safety module provides intelligent response measures and safety assurance for emergencies. The present invention allows passengers to independently control pick-up and drop-off points and adjust routes in real time through their mobile devices. With the support of a multimodal large model, it intelligently optimizes the pick-up and drop-off plan based on traffic conditions, road changes, and emergencies. This invention is particularly suitable for scenarios where passengers have flexible requirements for pick-up and drop-off points and can improve pick-up and drop-off efficiency and passenger experience in dynamically changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a system architecture diagram of an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0080] Figure 2This is a diagram of a passenger intelligent interaction control module of an adaptive pick-up and drop-off system for an autonomous driving vehicle based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0081] Figure 3 This is a multimodal large model optimization module diagram of an autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0082] Figure 4 This is a real-time path planning and dynamic adjustment module diagram of an autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0083] Figure 5 This is a diagram of a passenger feedback and personalized recommendation module of an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0084] Figure 6 This is a diagram of an intelligent accident response and safety module of an autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0085] Figure 7 1 is a schematic diagram of the A* algorithm flow of the autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0086] Figure 8 1 is a schematic diagram of the SENet structure of an adaptive pick-up and drop-off system for autonomous vehicles based on enhanced intelligent interaction provided by an embodiment of the present invention;

[0087] Figure 9 It is a schematic diagram of the CBAM structure of the autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The present invention will be described in detail below with reference to the accompanying drawings.

[0089] Example 1

[0090] The embodiment of the present invention provides an adaptive pick-up and drop-off system for autonomous driving vehicles based on enhanced intelligent interaction, such as Figure 1 As shown, it includes a passenger intelligent interaction control module, a multimodal large model optimization module, a real-time path planning and dynamic adjustment module, a passenger feedback and personalized recommendation module, and an intelligent accident response and safety module;

[0091] The passenger intelligent interactive control module is used to provide a real-time interactive interface between passengers and vehicles, allowing passengers to independently select, adjust or change pick-up points through their mobile devices and control the vehicle's driving path in real time;

[0092] The multimodal large model optimization module analyzes the behavior pattern of each passenger in the group through the multimodal large model, combines the current environmental data and historical behavior data, and predicts the future movement path and time schedule of each passenger in the group;

[0093] The real-time path planning and dynamic adjustment module is used to generate the real-time driving path of the vehicle and allow passengers to dynamically adjust the path during driving;

[0094] The passenger feedback and personalized recommendation module generates personalized pick-up point recommendations and optimization plans based on passengers' historical behavior, preferences, and feedback;

[0095] The intelligent accident response and safety module is used to provide intelligent response measures and safety assurance for emergencies;

[0096] The passenger intelligent interaction control module, the multimodal large model optimization module, the real-time path planning and dynamic adjustment module, the passenger feedback and personalized recommendation module and the intelligent accident response and safety module are interconnected.

[0097] Furthermore, the passenger intelligent interaction control module includes a pick-up point selection submodule, a route adjustment submodule and a preference setting submodule, such as Figure 2 As shown;

[0098] The pick-up point selection submodule is used for passengers to select from a plurality of predefined pick-up points or manually input a new pick-up point;

[0099] The route adjustment submodule is used for passengers to view the vehicle's route in real time and actively adjust the route in the event of road congestion or accidents;

[0100] The preference setting submodule is used to set the passenger's personalized pick-up and drop-off preferences through a decision tree model and generate recommendations, wherein the personalized pick-up and drop-off preferences at least include selecting a quieter pick-up and drop-off point, being close to a building entrance, and avoiding crowded areas;

[0101] The specific implementation process of the pick-up point selection submodule is as follows:

[0102] Use a deep learning model to parse the passenger's voice or text input to find the pick-up point. The specific process is as follows:

[0103] Voice input, using a Wave2Vec-based speech recognition model to convert speech into text;

[0104] Text parsing: Using the BERT model for semantic analysis to extract the location, time, and route requirements for riding;

[0105] The specific implementation process of the path adjustment submodule is as follows:

[0106] Route planning is performed based on the graph search A* algorithm. If the passenger changes the pick-up point or route, the optimal route is dynamically recalculated. The formula is:

[0107] f(n)=g(n)+h(n),

[0108] Among them, f(n) represents the total cost, g(n) represents the actual cost from the starting point to the current node, and h(n) represents the estimated cost from the current node to the target;

[0109] The specific implementation process of the preference setting submodule is as follows:

[0110] Feature fusion: Inputting passengers' immediate needs and long-term preferences as features into a decision tree model and training the model, wherein the immediate needs are current specific conditions, including at least weather, time, and location, and the long-term preferences represent passenger preference statistics based on historical data;

[0111] Weight setting: set different weights for different features, and set the weight of long-term preferences higher than the weight of immediate needs;

[0112] Dynamic adjustment: Dynamically adjust weight settings based on real-time feedback during decision tree model training;

[0113] Model evaluation and iteration: regularly use new data evaluation models to evaluate the decision tree model, and update the parameters of the decision tree model based on the evaluation results;

[0114] The model is deployed and used to prioritize passengers' different preferences using the trained decision tree model, and personalized pick-up and drop-off recommendations are made based on passenger historical records.

[0115] Furthermore, the multimodal large model optimization module includes a multimodal input processing submodule, a traffic and environment perception submodule and an intelligent optimization engine submodule, such as Figure 3 As shown;

[0116] Specifically, the process of interaction between the passenger intelligent interaction control module and the multimodal large model optimization module is as follows:

[0117] Interaction Description:

[0118] Input: The passenger intelligent interaction control module receives instructions from passengers, such as pick-up point selection, route adjustment, and preference settings;

[0119] Output: These instructions are used as input to the multimodal large model optimization module to convey passenger needs, such as text, voice, and images;

[0120] Algorithm model: Passenger instructions are parsed through a natural language processing model, including but not limited to BERT, and then integrated with real-time traffic environment data to generate appropriate pick-up points and route optimization plans through CLIP or multimodal Transformer models.

[0121] The multimodal input processing submodule is used to analyze the passenger's voice or text input through natural language processing technology to generate text input data; confirm the environmental image uploaded by the passenger through image analysis technology to generate image input data; and perform data fusion processing through the conditional variational autoencoding (CVAE) model to generate multimodal fusion data;

[0122] The traffic and environment perception submodule is used to obtain real-time traffic and environment data from traffic sensors, map APIs, and weather APIs. The real-time traffic and environment data is used to optimize the vehicle's driving path and pick-up points;

[0123] The intelligent optimization engine submodule is used to perform predictive analysis on the real-time traffic and environmental data through a multimodal large model, and to intelligently adjust the passenger pick-up and drop-off plan.

[0124] Furthermore, the specific implementation process of data fusion processing through the conditional variational autoencoding (CVAE) model in the multimodal input processing submodule is as follows:

[0125] Speech processing, using Wave2Vec to convert passenger speech into text data;

[0126] Image processing: For environmental images uploaded by passengers, DeepLabv3+ image semantic segmentation network is used to extract features, and the attention mechanism is integrated to enhance object recognition in environmental images to generate image semantic segmentation data;

[0127] It should be noted that the fusion attention mechanism adopts the channel attention mechanism SENet architecture and the spatial attention mechanism CBAM architecture;

[0128] The specific calculation process of SENet includes three steps: compression, expansion, and weighting. Figure 8 As shown:

[0129] a1. Through compression operation F sq The input feature map H∈R H*W*C Compressed into a one-dimensional vector z∈R with C channels through the global average pooling layer 1*1*C , F sq The calculation formula is:

[0130]

[0131] Among them, u c ∈RH*W Represents a pixel point on a feature channel in the feature map, C represents the number of channels, H represents the number of rows of the feature map, and W represents the number of columns of the feature map;

[0132] a2. Then perform the expansion operation F ex First, the obtained one-dimensional vector z is mapped to a one-dimensional vector with a channel number C′ through a fully connected layer, and then it is expanded back to a one-dimensional channel weight vector s∈R with a channel number C through a fully connected layer. 1*1*C , F ex The calculation formula is:

[0133]

[0134] Where C'=C / r, r represents the scaling factor. σ represents the Sigmoid activation function, δ represents the Relu activation function, and W2∈R C ' *C represents the fully connected layer, W1∈R C*C ' represents the fully connected layer;

[0135] a3.Finally, the weighted operation F scale Multiplying the one-dimensional channel weight vector s with the original input feature map H in the channel dimension will give the channel-weighted output feature map H'∈R H*W*C , F scale The calculation formula is:

[0136] H'=F scale (u c ,s c )=u c .s c ,

[0137] Among them, s c ∈R C Represents the weight of a channel;

[0138] SENet focuses on channels with greater feature contribution and is an effective method for filtering redundant feature channel information. It performs feature filtering on environmental images uploaded by passengers, filters out invalid redundant features, and improves feature extraction efficiency.

[0139] The specific calculation process of CBAM includes three steps: compression, fusion, and weighting. Figure 9 As shown:

[0140] b1. Input feature map H∈R H*W*C Perform the maximum pooling operation F on the channel dimension respectively max and average pooling operation F avg Get the two-dimensional vector z1∈R H*W*1 and z2∈R H*W*1 , F maxand F avg The calculation formula is:

[0141] z1=F max (v c )=max(v c ),

[0142]

[0143] Among them, v c ∈R C Represents a pixel point at a certain pixel position in the feature map;

[0144] b2. Fuse z1 and z2 cat , that is, stacking and fusion are performed on the channel dimension to obtain z m ∈R H*W*2 , then z m Use 7×7 convolution to reduce the dimension of F conv Get the spatial two-dimensional weight vector s∈R H*W*1 , F max and F avg The calculation formula is:

[0145] z m =F cat (z1,z2)=cat(z1,z1),

[0146] s=F conv (z m )=f 7×7 (z m ),

[0147] Among them, f 7×7 Indicates that the convolution kernel is 7;

[0148] b3. Through weighted operation F scale Multiply the spatial two-dimensional weight s with the corresponding position of the original input feature map H in the spatial dimension to obtain the output feature map H'∈R after spatial dimension weighting H*W*C , fuse z1 and z2 cat , that is, stacking and fusion are performed on the channel dimension to obtain z m ∈R H*W*2 , F scale The calculation formula is:

[0149] H'=F scale (s c ,v c )=s c .v c ,

[0150] Among them, s c ∈R H*WRepresents the weight of a pixel position on the feature map;

[0151] The CBAM attention mechanism is a good combination of channel attention mechanism and spatial attention mechanism to enhance the feature extraction of environmental images.

[0152] Text parsing: Using the GPT-4 model to perform semantic analysis on passenger text input and generate text semantic parsing data;

[0153] The conditional variational autoencoding (CVAE) model embeds the text data, the image semantic segmentation data, and the text semantic parsing data into the same vector space, and realizes multimodal fusion by maximizing the similarity between related modalities.

[0154] The specific implementation process of the intelligent optimization engine submodule for predicting and analyzing the real-time traffic and environmental data through a multimodal large model and intelligently adjusting the passenger pick-up and drop-off plan is as follows:

[0155] The multimodal large model includes a time encoding model, a space encoding model, two conditional variational autoencoding (CVAE) models, a fusion decoding model and a semantic BEV modal prediction model. The time encoding model takes the historical behavior data of the target passenger to be predicted as input and generates time encoding. The space encoding model takes the multimodal fusion data as input and generates space encoding. The time encoding model and the space encoding model will generate time-space features. The time encoding model adopts a bidirectional long short-term memory (LSTM) network, and the space encoding model adopts a DenseNet network. The two conditional variational autoencoding (CVAE) models include a first CVAE model and a second CVAE model. The output of the time encoding model is connected to the input of the first CVAE model. The output of the space encoding model is connected to the input of the second CVAE model. The fusion decoding model converts the connected time-space features into a prediction trajectory with confidence. The semantic BEV modal prediction model is used to optimize the confidence of the prediction trajectory.

[0156] The real-time prediction process is:

[0157] Decomposing the entire prediction task into two sub-models: a modal prediction model TPM based on modal distribution and a trajectory prediction model MPM based on modal trajectory distribution, both of which are based on the multimodal large model;

[0158] Given historical passenger behavior data and real-time traffic and environmental data, TPM performs probabilistic inference on predicted trajectories for all modalities;

[0159] The trajectory prediction model (MPM) of the modal trajectory distribution is used to solve the comprehensive probability of different modes and compare it with a preset probability threshold to determine whether the comprehensive probability is greater than the preset probability threshold. If so, the passenger pick-up and drop-off plan is adjusted. At the same time, the semantic BEV modal prediction model is used to optimize the confidence level of each modal trajectory prediction.

[0160] Furthermore, the real-time path planning and dynamic adjustment module includes a real-time path planning submodule and a dynamic adjustment and optimization submodule, such as Figure 4 As shown;

[0161] The real-time route planning submodule generates an optimal driving route based on the passenger's currently selected pick-up point and real-time traffic conditions, and recalculates the route for each change, including changes to the pick-up point and real-time traffic conditions;

[0162] The dynamic adjustment and optimization submodule is used to generate an alternative route when an emergency occurs to the vehicle, and adjust the alternative route according to the feedback from the passengers to generate a final driving route.

[0163] Furthermore, the specific implementation process of the real-time path planning submodule is as follows:

[0164] Combine multi-sensor data and use SLAM technology for environmental mapping and self-positioning to achieve dynamic path adjustment;

[0165] Based on the passenger's selected pick-up point and real-time traffic data, the A* algorithm is used to generate the optimal path. The path planning formula uses the heuristic function:

[0166]

[0167] Where H(n) represents the estimated cost from the current node n to the target node, (x1, y1) represents the coordinates of the current node, and (x2, y2) represents the coordinates of the target node;

[0168] Specifically, in the A* algorithm, there are two important sets: Open list and Closed list. Points that have not been detected during the path planning process are stored in the Open list, and points that have been detected (or passed through) are stored in the Closed list. Points in the Open list may not necessarily be passed through in the future planned path. First, put the starting point A into the Open list, and check the reachable points around the starting point. This embodiment is set as a nearby point, put these points into the Open list, set the starting point as the parent node of these points, and move the starting point from the Open list to the Closed list. Then, the algorithm will select a point B in the Open list with the smallest F(n) value between it and the starting point, and use point B as the current processing node, and repeat the above operations until the destination node is added to the Open list. The specific process is as follows: Figure 7 As shown;

[0169] In the present invention, for a vehicle, the center position coordinates (x, y) of the path point are used. According to such rules, the networkx toolkit is used to establish the traffic road into a topological graph structure, so that the traffic road is abstracted into the form of a topological graph. The A* algorithm is used to search for multiple path point sequences from the current point to the global target, and the sequences are arranged according to the distance. The shortest path is taken as the target path.

[0170] The specific process of the dynamic adjustment and optimization submodule is as follows:

[0171] When an emergency event is detected, a new path plan is generated using a genetic algorithm. The genetic algorithm generates a path population through crossover, mutation, and selection operations, scores the adaptability of each path, and selects the optimal path.

[0172] The alternative path generation formula is:

[0173] C(x,y)=min[C(x-1,y),C(x,y-1)]+w(x,y),

[0174] Among them, C(x,y) represents the minimum cost from the starting point to the current node (x,y), and w(x,y) represents the weight of the current node.

[0175] Furthermore, the passenger feedback and personalized recommendation module includes a personalized recommendation submodule and a real-time feedback submodule, such as Figure 5 As shown;

[0176] Specifically, the passenger feedback and personalized recommendation module interacts with the multimodal large model optimization module as follows:

[0177] Interaction Description:

[0178] Input: The passenger feedback module obtains information from passengers’ historical behavior, preferences, and real-time feedback;

[0179] Output: This feedback information is passed to the multimodal large model optimization module for personalized recommendation of pick-up points and routes;

[0180] Algorithm model: Generate personalized pick-up point recommendations for passengers through collaborative filtering and matrix decomposition technology.

[0181] The personalized recommendation submodule is used to prioritize the passenger's preferred pick-up points and routes based on the passenger's historical data, wherein the historical data includes previously used pick-up points and previously preferred routes;

[0182] The real-time feedback submodule is used to provide passengers with the vehicle's real-time location, estimated arrival time, and route suggestions via their smartphones, and to adjust the pick-up and drop-off plan in real time based on passenger feedback;

[0183] The specific implementation process of the personalized recommendation submodule is as follows:

[0184] Collaborative filtering technology is used to analyze passengers' historical pick-up and drop-off data, and personalized recommendations are achieved through a matrix decomposition model. The formula is:

[0185] R=P*Q T ,

[0186] Among them, R represents the rating matrix of the pick-up point, P and Q are the feature matrices of passengers and pick-up points respectively. By optimizing P and Q, we can predict the passenger's rating of the recommended pick-up point.

[0187] Furthermore, the intelligent accident response and safety module includes an accident response submodule and a safety priority mechanism submodule, such as Figure 6 As shown;

[0188] Specifically, the interaction process between the real-time path planning and dynamic adjustment module and the intelligent accident response and safety module is as follows:

[0189] Interaction Description:

[0190] Input: The real-time path planning module perceives emergencies through vehicle sensors and external data, such as road conditions and accident data;

[0191] Output: The real-time path planning module generates alternative paths and notifies the intelligent accident response module;

[0192] Algorithm model: When a road closure or traffic accident is detected, the system uses Bayesian networks and Markov decision processes to calculate a safe path and the optimal response plan.

[0193] The accident response submodule is used to automatically generate an alternative route and notify passengers to adjust their routes when a road accident or road closure is detected;

[0194] The safety priority mechanism submodule prioritizes the safest pick-up and drop-off plan based on real-time traffic and environmental data. Furthermore, the specific implementation process of the accident response submodule is as follows:

[0195] If a road accident or road closure is detected, the Bayesian network is used to calculate the probabilities of different response plans and select the optimal alternative path. The formula is:

[0196]

[0197] Where P(H|E) represents the probability that hypothesis H holds given that event E has occurred, P(E|H) represents the probability that event E will occur under hypothesis H, and P(H) and P(E) represent the prior probabilities of hypothesis H and time E.

[0198] The specific implementation process of the safety priority mechanism submodule is as follows:

[0199] The Markov decision process (MDP) is used to calculate the risk of each state based on environmental data and select the solution with the highest security. The Markov decision formula is:

[0200] V(s)=max a [R(s,a)+b∑ s' P(s'|s,a)V(s'))],

[0201] Where V(s) represents the value of state s, R(s,a) represents the immediate reward of executing action a in state s, b represents the discount factor, and P(s'|s,a) represents the transition probability from state s to state s'.

[0202] Example 2: Real-time route adjustment for passengers in busy streets

[0203] 1. Background: A passenger is in a busy urban area, and the autonomous vehicle encounters a traffic jam while picking them up. The passenger wishes to adjust the pick-up plan using their smartphone.

[0204] 2. Passenger operation:

[0205] The passenger uses the smartphone app to view the vehicle's real-time route and discovers that the road ahead is congested due to an accident. The passenger manually selects an alternative pickup point in the app and confirms the updated route.

[0206] 3. System Optimization: The system recalculates new pick-up and drop-off routes using a multimodal large-scale model and generates an optimal driving plan. The updated vehicle arrival time is also pushed to passengers via the mobile app.

[0207] 4. Dynamic feedback: The vehicle arrives at the passenger’s designated alternative pick-up point based on the new route, ensuring a smooth and error-free pick-up process.

[0208] Example 3: Intelligent path switching under emergency conditions

[0209] 1. Background: A passenger was on their way to a conference center when their original route encountered a sudden road closure. The system automatically detected the situation and generated an alternative route.

[0210] 2. System operation:

[0211] The system detects a road closure ahead and immediately generates an alternative route, notifying the passenger to confirm the new pick-up point through the app. The system analyzes the current traffic conditions using a multimodal large model to select the optimal pick-up point, avoiding congested areas.

[0212] 3. Passenger confirmation: The passenger confirms the new pick-up plan through the intelligent interactive interface, and the vehicle proceeds to the new pick-up point according to the adjusted route to ensure that the passenger arrives on time.

[0213] Example 4: Personalized Pickup Point Preference Management

[0214] 1. Background: A passenger is accustomed to getting on and off the bus at a quiet neighborhood entrance. The system learns this passenger's pickup and drop-off preferences and provides personalized recommendations for future pickups.

[0215] 2. System operation:

[0216] The system analyzes the passenger's historical pick-up and drop-off records and prioritizes pick-up and drop-off points that are quieter and closer to the building entrance. Passengers can view the system's recommended pick-up and drop-off options through the app and confirm or adjust the final pick-up and drop-off point.

[0217] 3. Dynamic Optimization: Based on traffic and environmental changes, the system dynamically optimizes routes during the pick-up and drop-off process to ensure that the passenger's pick-up and drop-off experience meets their preferences.

[0218] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the best mode of the present invention.

[0219] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0220] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0221] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0222] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0223] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

Claims

1. An adaptive pick-up and drop-off system for autonomous vehicles based on enhanced intelligent interaction, characterized by: It includes a passenger intelligent interaction control module, a multimodal large model optimization module, a real-time path planning and dynamic adjustment module, a passenger feedback and personalized recommendation module, and an intelligent accident response and safety module; The passenger intelligent interactive control module is used to provide a real-time interactive interface between passengers and vehicles, allowing passengers to independently select, adjust or change pick-up points through their mobile devices and control the vehicle's driving path in real time; The multimodal large model optimization module analyzes the behavior pattern of each passenger in the group through the multimodal large model, combines the current environmental data and historical behavior data, and predicts the future movement path and time schedule of each passenger in the group; The real-time path planning and dynamic adjustment module is used to generate the real-time driving path of the vehicle and allow passengers to dynamically adjust the path during driving; The passenger feedback and personalized recommendation module generates personalized pick-up point recommendations and optimization plans based on passengers' historical behavior, preferences, and feedback; The intelligent accident response and safety module is used to provide intelligent response measures and safety assurance for emergencies; The passenger intelligent interaction control module, the multimodal large model optimization module, the real-time path planning and dynamic adjustment module, the passenger feedback and personalized recommendation module, and the intelligent accident response and safety module are interconnected; The multimodal large model optimization module includes a multimodal input processing submodule, a traffic and environment perception submodule, and an intelligent optimization engine submodule; The multimodal input processing submodule is used to analyze the passenger's voice or text input through natural language processing technology to generate text input data; confirm the environmental image uploaded by the passenger through image analysis technology to generate image input data; and perform data fusion processing through the conditional variational autoencoding (CVAE) model to generate multimodal fusion data; The traffic and environment perception submodule is used to obtain real-time traffic and environment data from traffic sensors, map APIs, and weather APIs. The real-time traffic and environment data is used to optimize the vehicle's driving path and pick-up points; The intelligent optimization engine submodule is used to predict and analyze the real-time traffic and environmental data through a multimodal large model, and intelligently adjust the passenger pick-up and drop-off plan; The specific implementation process of data fusion processing through the conditional variational autoencoding (CVAE) model in the multimodal input processing submodule is as follows: Speech processing, using Wave2Vec to convert passenger speech into text data; Image processing: For environmental images uploaded by passengers, DeepLabv3+ image semantic segmentation network is used to extract features, and the attention mechanism is integrated to enhance object recognition in environmental images to generate image semantic segmentation data; Text parsing: Using the GPT-4 model to perform semantic analysis on passenger text input and generate text semantic parsing data; The conditional variational autoencoding (CVAE) model embeds the text data, the image semantic segmentation data, and the text semantic parsing data into the same vector space, and realizes multimodal fusion by maximizing the similarity between related modalities. The specific implementation process of the intelligent optimization engine submodule for predicting and analyzing the real-time traffic and environmental data through a multimodal large model and intelligently adjusting the passenger pick-up and drop-off plan is as follows: The multimodal large model includes a time encoding model, a space encoding model, two conditional variational autoencoding (CVAE) models, a fusion decoding model and a semantic BEV modal prediction model. The time encoding model takes the historical behavior data of the target passenger to be predicted as input and generates time encoding. The space encoding model takes the multimodal fusion data as input and generates space encoding. The time encoding model and the space encoding model will generate time-space features. The time encoding model adopts a bidirectional long short-term memory (LSTM) network, and the space encoding model adopts a DenseNet network. The two conditional variational autoencoding (CVAE) models include a first CVAE model and a second CVAE model. The output of the time encoding model is connected to the input of the first CVAE model. The output of the space encoding model is connected to the input of the second CVAE model. The fusion decoding model converts the connected time-space features into a prediction trajectory with confidence. The semantic BEV modal prediction model is used to optimize the confidence of the prediction trajectory. The real-time prediction process is: Decomposing the entire prediction task into two sub-models: a modal prediction model TPM based on modal distribution and a trajectory prediction model MPM based on modal trajectory distribution, both of which are based on the multimodal large model; Given historical passenger behavior data and real-time traffic and environmental data, TPM performs probabilistic inference on predicted trajectories for all modalities; The trajectory prediction model (MPM) of the modal trajectory distribution is used to solve the comprehensive probability of different modes and compare it with a preset probability threshold to determine whether the comprehensive probability is greater than the preset probability threshold. If so, the passenger pick-up and drop-off plan is adjusted. At the same time, the semantic BEV modal prediction model is used to optimize the confidence level of each modal trajectory prediction.

2. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 1 is characterized in that: The passenger intelligent interaction control module includes a pick-up point selection submodule, a route adjustment submodule and a preference setting submodule; The pick-up point selection submodule is used for passengers to select from a plurality of predefined pick-up points or manually input a new pick-up point; The route adjustment submodule is used for passengers to view the vehicle's route in real time and actively adjust the route in the event of road congestion or accidents; The preference setting submodule is used to set the passenger's personalized pick-up and drop-off preferences through a decision tree model and generate recommendations, wherein the personalized pick-up and drop-off preferences at least include selecting a quieter pick-up and drop-off point, being close to a building entrance, and avoiding crowded areas; The specific implementation process of the pick-up point selection submodule is as follows: Use a deep learning model to parse the passenger's voice or text input to find the pick-up point. The specific process is as follows: Voice input, using a Wave2Vec-based speech recognition model to convert speech into text; Text parsing: Using the BERT model for semantic analysis to extract the location, time, and route requirements for riding; The specific implementation process of the path adjustment submodule is as follows: Route planning is performed based on the graph search A* algorithm. If the passenger changes the pick-up point or route, the optimal route is dynamically recalculated. The formula is: f(n)=g(n)+h(n), Among them, f(n) represents the total cost, g(n) represents the actual cost from the starting point to the current node, and h(n) represents the estimated cost from the current node to the target; The specific implementation process of the preference setting submodule is as follows: Feature fusion: Inputting passengers' immediate needs and long-term preferences as features into a decision tree model and training the model, wherein the immediate needs are current specific conditions, including at least weather, time, and location, and the long-term preferences represent passenger preference statistics based on historical data; Weight setting: set different weights for different features, and set the weight of long-term preferences higher than the weight of immediate needs; Dynamic adjustment: Dynamically adjust weight settings based on real-time feedback during decision tree model training; Model evaluation and iteration: regularly use new data evaluation models to evaluate the decision tree model, and update the parameters of the decision tree model based on the evaluation results; The model is deployed and used to prioritize passengers' different preferences using the trained decision tree model, and personalized pick-up and drop-off recommendations are made based on passenger historical records.

3. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 2 is characterized in that: The real-time path planning and dynamic adjustment module includes a real-time path planning submodule and a dynamic adjustment and optimization submodule; The real-time route planning submodule generates an optimal driving route based on the passenger's currently selected pick-up point and real-time traffic conditions, and recalculates the route for each change, including changes to the pick-up point and real-time traffic conditions; The dynamic adjustment and optimization submodule is used to generate an alternative route when an emergency occurs to the vehicle, and adjust the alternative route according to the feedback from the passengers to generate a final driving route.

4. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 3 is characterized in that: The specific implementation process of the real-time path planning submodule is as follows: Combine multi-sensor data and use SLAM technology for environmental mapping and self-positioning to achieve dynamic path adjustment; Based on the passenger's selected pick-up point and real-time traffic data, the A* algorithm is used to generate the optimal path. The path planning formula uses the heuristic function: Where H(n) represents the estimated cost from the current node n to the target node, (x1, y1) represents the coordinates of the current node, and (x2, y2) represents the coordinates of the target node; The specific process of the dynamic adjustment and optimization submodule is as follows: When an emergency event is detected, a new path plan is generated using a genetic algorithm. The genetic algorithm generates a path population through crossover, mutation, and selection operations, scores the adaptability of each path, and selects the optimal path. The alternative path generation formula is: C(x,y)=min[C(x-1,y),C(x,y-1)]+w(x,y), Among them, C(x,y) represents the minimum cost from the starting point to the current node (x,y), and w(x,y) represents the weight of the current node.

5. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 4 is characterized in that: The passenger feedback and personalized recommendation module includes a personalized recommendation submodule and a real-time feedback submodule; The personalized recommendation submodule is used to prioritize the passenger's preferred pick-up points and routes based on the passenger's historical data, wherein the historical data includes previously used pick-up points and previously preferred routes; The real-time feedback submodule is used to provide passengers with the vehicle's real-time location, estimated arrival time, and route suggestions via their smartphones, and to adjust the pick-up and drop-off plan in real time based on passenger feedback; The specific implementation process of the personalized recommendation submodule is as follows: Collaborative filtering technology is used to analyze passengers' historical pick-up and drop-off data, and personalized recommendations are achieved through a matrix decomposition model. The formula is: R=P*Q T , Among them, R represents the rating matrix of the pick-up point, P and Q are the feature matrices of passengers and pick-up points respectively. By optimizing P and Q, we can predict the passenger's rating of the recommended pick-up point.

6. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 5 is characterized in that: The intelligent accident response and safety module includes an accident response submodule and a safety priority mechanism submodule; The accident response submodule is used to automatically generate an alternative route and notify passengers to adjust their routes when a road accident or road closure is detected; The safety priority mechanism submodule prioritizes the safest pick-up and drop-off plan based on real-time traffic and environmental data.

7. The autonomous driving vehicle adaptive pick-up and drop-off system based on enhanced intelligent interaction according to claim 6 is characterized in that: The specific implementation process of the accident response submodule is as follows: If a road accident or road closure is detected, the Bayesian network is used to calculate the probabilities of different response plans and select the optimal alternative path. The formula is: Where P(H|E) represents the probability that hypothesis H holds given that event E has occurred, P(E|H) represents the probability that event E will occur under hypothesis H, and P(H) and P(E) represent the prior probabilities of hypothesis H and time E. The specific implementation process of the safety priority mechanism submodule is as follows: The Markov decision process (MDP) is used to calculate the risk of each state based on environmental data and select the solution with the highest security. The Markov decision formula is: V(s)=max a [R(s,a)+b∑ s' P(s'|s,a)V(s'))], Among them, V(s) represents the value of state s, R(s,a) represents the immediate reward of state s performing action a, b represents the discount factor, and P(s ' |s,a) means from state s to state s ' The transition probability.

Images