Automatic driving vehicle pick-up point generation system based on space mapping and environment remodeling

Abstract

The invention provides an automatic driving vehicle pick-up point generation system based on space mapping and environment remodeling, which belongs to the technical field of automatic driving and comprises a space sensing and mapping module S100, a virtual pick-up area generation module S200, a personalized environment remodeling module S300, an augmented reality AR and guiding module S400 and an intelligent safety and dynamic optimization module S500. A virtual pick-up area generated by the system is generated based on multi-modal data fusion, space environment mapping and a deep Q network, and the position of a generated virtual pick-up point is closer to the actual position; the surrounding environment can be sensed and mapped in real time, the virtual pick-up area is dynamically generated, the generated virtual pick-up area not only can be automatically generated according to environment changes, but also can be visually displayed through technologies such as virtual marks and AR display, and passengers and vehicles are helped to achieve accurate docking in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a system for generating pick-up and drop-off points for autonomous driving vehicles based on space mapping and environment reconstruction. Background Art

[0002] In scenarios such as crowded urban environments, festivals, large-scale events or emergencies, traditional pick-up points may be inaccessible due to road closures, temporary obstacles or crowds. Existing autonomous vehicle pick-up systems usually rely on fixed pick-up points to cope with these dynamic changes, and are unable to flexibly adjust or create new pick-up areas. Even if some systems can temporarily change the pick-up points, they lack visual guidance, making it difficult for passengers to accurately find the vehicle.

[0003] The present invention proposes a brand-new solution. Combining spatial mapping and augmented reality technology, the system can perceive and map the surrounding environment in real time and dynamically generate virtual pick-up areas. The generated virtual pick-up areas can not only be automatically generated according to environmental changes, but also be visualized through virtual signs and AR display technologies, helping passengers and vehicles to achieve accurate docking in complex environments. This technology is particularly suitable for the intelligent generation and adjustment of pick-up points in busy urban areas, temporary activities and traffic emergencies. Summary of the invention

[0004] The purpose of the present invention is to overcome the above problems existing in the prior art and to greatly improve the technical effect thereof on the basis of the original technology; for this purpose, the present invention provides a system for generating pick-up and drop-off points for autonomous driving vehicles based on spatial mapping and environmental reconstruction, the system comprising:

[0005] Spatial perception and mapping module S100; virtual pick-up area generation module S200; personalized environment reshaping module S300; augmented reality AR and guidance module S400; intelligent safety and dynamic optimization module S500.

[0006] Among them, the spatial perception and mapping module S100 is used to perceive the surrounding environment through multimodal sensors, including: LiDAR, camera and GPS, generate a three-dimensional spatial model, and identify areas suitable for generating virtual pick-up points; the spatial perception and mapping module includes: real-time spatial mapping and dynamic environmental analysis; the real-time spatial mapping generates a three-dimensional spatial model through real-time environmental perception, and identifies buildings, streets, pedestrian flow areas and traffic jams through the generated three-dimensional spatial model, providing a basis for the generation of virtual pick-up areas; the dynamic environmental analysis will include: combining traffic data, weather information and temporary events to dynamically adjust the three-dimensional spatial model to ensure the safety and accessibility of the virtual pick-up points.

[0007] Specifically, the implementation of the spatial perception and mapping module S100 includes: multimodal data fusion, deep learning environment reconstruction and dynamic scene understanding; wherein, the multimodal data fusion uses the extended Kalman filter EKF for multimodal data fusion; the multimodal data is obtained through sensor input, and the sensor input includes: LiDAR, camera and GPS input data; the LiDAR captures high-precision three-dimensional point cloud data of the surrounding environment; the camera obtains information including roads, pedestrians, vehicles and buildings through image processing technology; the GPS provides vehicle positioning and motion information; the EKF combines the obtained sensor data to obtain the precise position and posture of the target in the environment, and updates it in real time; the target state update formula is:

[0008] S t =F(S t－1 , Z t , U t )+W t

[0009] Among them, S t is the target state at the current moment, F is the state transfer matrix, S t－1 is the target state at the previous moment, Z t is the sensor measurement value vector, U t is the control input, W t is the process noise.

[0010] Specifically, deep learning environment reconstruction includes: on the basis of obtaining the precise position and posture information of the target in the environment by EKF, using convolutional neural network CNN combined with graph neural network GNN to perform deep learning to process multimodal data, identify the key features of the target at the corresponding position in the environment, and establish the connection between different targets; including identifying: obstacles, pedestrians and traffic lights, and generating a refined three-dimensional model; the steps of generating a three-dimensional model are: S101, fusing point cloud and image data: fusing the point cloud model fused by EKF and the depth map extracted by CNN to generate a more accurate three-dimensional point cloud; S102, performing three-dimensional reconstruction: generating a three-dimensional geometric model of the environment through a three-dimensional reconstruction algorithm triangular mesh based on the generated three-dimensional point cloud, and the three-dimensional geometric model includes the shape and position of all perceived targets; then, using GNN to model the spatial relationship between different targets and further optimize the model; S103, optimizing the obtained three-dimensional model: modeling the interaction between different targets based on GNN, further optimizing the three-dimensional model, optimizing the relative position and shape of the target, and ensuring the geometric accuracy and semantic consistency of the model; and using GN N optimizes the entire three-dimensional model to ensure the accuracy and integrity of the model; S104, generates a refined three-dimensional model: through steps S101, S102 and S103 combined with the dynamic estimation of EKF, a global refined three-dimensional model is generated through GNN.

[0011] Specifically, dynamic scene understanding includes: detecting and identifying objects in dynamic scenes in real time by integrating YOLOv5, including identifying pedestrians and vehicles in motion; using a time series analysis model based on a temporal convolutional network (TCN) to predict traffic flow and pedestrian flow dynamics; the steps of dynamic scene understanding are: S201, first, processing each video frame in the dynamic scene by YOLOv5, identifying the target and the position of the target in each video frame, wherein the target includes pedestrians, vehicles and obstacles; S202, arranging the target detection results output by YOLOv5 in chronological order and using them as inputs of a TCN model, and training the TCN model to understand the behavior of the target in time sequence; S203, the trained TCN model further optimizes the tracking of the object by capturing the motion pattern of the object; at the same time, while identifying the motion trajectory of the target, the motion state of the target is predicted, and the system makes different decisions for the dynamic scene according to the different motion states and motion patterns of the target, wherein the different decisions include: honking, stopping and decelerating.

[0012] Among them, the virtual pick-up area generation module S200 is used to dynamically generate a virtual pick-up area based on the spatial perception and mapping module S100; the generated area not only includes establishing suitable pick-up points in areas with convenient transportation, but also includes temporarily creating pick-up points in urban congested areas and activity venues, and adjusting according to passenger needs and environmental changes; the virtual pick-up area generation module S200 includes: virtual pick-up point creation and pick-up area adjustment; the virtual pick-up point creation is based on the spatial mapping results generated by the spatial perception and mapping module S100, and when there is no fixed pick-up point and the existing pick-up point is unreachable, a virtual pick-up point suitable for pick-up is generated; the pick-up area adjustment is used when the environment changes, the system automatically adjusts the pick-up area, generates a new virtual pick-up point, and notifies passengers and vehicles in real time.

[0013] Specifically, the virtual pick-up area generation module S200 dynamically generates an optimal virtual pick-up area according to real-time spatial perception data; the implementation of the virtual pick-up area generation module includes: optimal pick-up point generation, multi-objective optimization and dynamic adjustment mechanism; wherein, the optimal pick-up point generation includes: dynamically generating the pick-up area using a deep Q network in deep reinforcement learning technology; the steps of dynamically generating the pick-up area using the deep Q network are: S301, extracting historical multi-modal fused environmental perception data to form an environmental perception vector X i , i represents the i-th group of environmental perception vectors; S302, extracting the decision Y of the optimal pick-up point corresponding to the environmental perception vector i ; S303, X i As the input of the deep Q network, the corresponding Y i The deep Q network is trained as the output of the deep Q network. During the training process, the model continuously optimizes the location of the pick-up point according to passenger needs, environmental restrictions and safety constraints to obtain a trained deep Q network model, wherein the trained deep Q network model refers to a difference between the model predicted Q value and the target Q value that meets a preset condition; S304, extracting multimodal data information of the autonomous driving vehicle in real time, processing the multimodal data and inputting it into the trained deep Q network model, and judging the optimal pick-up point according to the output result; the decision Y of the optimal pick-up point corresponding to the environmental perception vector is extracted iThe steps are as follows: S401, dividing the entire image area corresponding to the environment perception vector into grids with horizontal and vertical lines; S402, determining whether each grid meets the pick-up and drop-off requirements according to the environment perception vector, if the corresponding grid includes pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are not met; if the corresponding grid does not include pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are met; S403, extracting an area composed of continuous grids that meet the pick-up and drop-off requirements in the entire image area as an alternative pick-up and drop-off area, and judging whether the shape and size of the autonomous driving vehicle can be placed in the alternative pick-up and drop-off area according to the shape and size of the autonomous driving vehicle; the area composed of continuous grids that meet the pick-up and drop-off requirements means that each grid in the alternative pick-up and drop-off area meets the pick-up and drop-off requirements, and each grid that meets the pick-up and drop-off requirements meets the pick-up and drop-off requirements. The grid to be delivered is at least continuous with another grid that meets the pick-up requirements, and the grids that meet the pick-up requirements in the candidate pick-up area formed must be filled with grids that meet the pick-up requirements; S404, select the candidate pick-up area whose shape and size can be placed in the autonomous driving vehicle as the feasible pick-up area, and determine the optimal pick-up point according to the shape regularity, area, safety and whether the vehicle is allowed to enter the feasible pick-up area; set all grid values ​​in the candidate pick-up area corresponding to the optimal pick-up point to 1, set all values ​​of other grids in the entire image area to 0, set the values ​​set for the horizontal rows of the grids in the entire image area as the values ​​of the horizontal rows of the matrix, and set the values ​​set for the vertical rows of the grids in the entire image area as the values ​​of the vertical rows of the matrix, to form the decision matrix Y of the optimal pick-up point in the entire image area i .

[0014] The multi-objective optimization includes: using a multi-objective optimization algorithm with a Pareto optimal solution to weigh the safety, passenger convenience and arrival time of different feasible pick-up areas to ensure the optimal location of the pick-up point; the dynamic adjustment mechanism includes: when the environment changes, the system updates the location of the pick-up point in real time through strategy adjustment based on Markov decision MDP.

[0015] Among them, the personalized environment reshaping module S300 reshapes the virtual pick-up area according to the personalized needs of each passenger, so that the reshaped virtual pick-up area meets the special requirements of the passengers; the personalized environment reshaping module S300 includes: personalized pick-up demand matching and dynamic reshaping of the environment; personalized pick-up demand matching is used to give priority to matching the personalized needs of passengers when generating the virtual pick-up area based on the historical behavior and preferences of passengers; dynamic reshaping of the environment is used to make personalized adjustments to the virtual pick-up point according to the passenger input needs and current environmental conditions, ensuring that the vehicle can meet the passengers at the optimal location.

[0016] Specifically, the personalized environment reshaping module S300 includes: generating and adjusting virtual pick-up points according to personalized needs of passengers; the implementation of the personalized environment reshaping module includes: personalized demand matching and personalized environment reshaping; wherein personalized demand matching includes: analyzing the historical behavior and preferences of passengers based on the recommendation system algorithm of Collaborative Filtering, and generating virtual pick-up areas that meet the needs of passengers; the recommendation system algorithm based on Collaborative Filtering extracts implicit passenger behavior patterns from massive passenger data through matrix decomposition technology, and generates personalized virtual pick-up points under passenger needs by applying conditional generative adversarial network CGAN, and the conditional input of the conditional generative adversarial network CGAN includes passenger preferences and current environmental restrictions; the personalized environment reshaping includes: dynamically adjusting the location of the virtual pick-up point through the Bayesian optimization algorithm to ensure that the pick-up point meets the personalized needs of passengers, especially in special locations including building entrances, parking lots and away from noise areas; the recommendation system algorithm of Collaborative Filtering performs score prediction based on the needs, historical behaviors and preferences of passengers, and predicts the pick-up points that meet the needs of passengers. The collaborative filtering score prediction formula is:

[0017]

[0018] in, represents the predicted score, which represents the score of passenger u for pick-up point i; μ is the global bias, b u and b i denote the offsets of the passenger and the pick-up point, respectively, q i and p u are the hidden vectors of the pick-up point and the passenger respectively, and T is q i The transpose of .

[0019] The CGAN generation model is: P(x|c)=G(z,c), where z is random noise, c is the passenger's personalized demand, x is the pick-up point location, G is the model generator, and P is the probability distribution of the pick-up point location x under given conditions c.

[0020] Among them, the augmented reality AR and guidance module S400 provides visual guidance for passengers through augmented reality AR technology, helping passengers to quickly and accurately find the virtual pick-up area; the augmented reality AR and guidance module S400 includes: AR signs and guidance and real-time AR feedback; the AR signs and guidance generate virtual pick-up signs through the passenger's mobile device, and guide the passenger from the current location to the virtual pick-up sign through the AR interface; the real-time AR feedback means that when the passenger approaches the pick-up area, the system will provide real-time feedback through the AR interface to guide the passenger to accurately locate the vehicle.

[0021] Specifically, the augmented reality AR and guidance module S400 includes: providing visual guidance for passengers through AR technology to help them reach the pick-up area quickly and accurately; the implementation of the augmented reality AR and guidance module S400 includes: AR visual navigation and real-time path guidance; wherein, AR visual navigation includes: using SLAM technology to implement AR navigation on the passenger's mobile device; the SLAM algorithm scans the environment in real time through the device camera, builds a map of the surrounding environment, and locates the position of the passenger's mobile device; combined with the deep learning visual feature extraction network ResNet50 to detect surrounding objects, and generate virtual pick-up point signs based on the identified surrounding key objects and areas; through the AR system, the surrounding environment is projected in real time and the path to the pick-up point is planned. The passenger views the generated virtual pick-up point sign and the generated path plan through the mobile device, and preliminarily generates the path from the passenger to the pick-up point; the real-time path guidance includes: based on the path planning algorithm Dijkstra algorithm combined with the obstacle information in the AR system, dynamically adjust the path from the passenger to the pick-up point; at the same time, the system will display dynamic signs through the AR interface when the passenger approaches the pick-up point, and provide accurate navigation instructions; the SLAM navigation equation is:

[0022] x t =f(x t－1 ,u t )+w t

[0023] Among them, x t is the passenger’s current position, x t－1 The passenger's position at the last moment, f(x t－1 ,u t ) is the state transfer function, u t is the control input, w t is the process noise.

[0024] Among them, the intelligent safety and dynamic optimization module S500 is used to ensure the safety and dynamic optimization of autonomous driving vehicles while the system generates a virtual pick-up area, and to avoid generating pick-up points in dangerous and inaccessible areas; the intelligent safety and dynamic optimization module S500 includes: safety priority generation and dynamic adjustment and feedback; safety priority generation means that when the system generates a virtual pick-up area, it combines: pedestrian flow data, traffic conditions and temporary obstacle information to ensure the safety of the pick-up area and avoid generating pick-up points on highways and congested areas; dynamic adjustment and feedback are used when the environment changes, the system will automatically adjust the virtual pick-up area, and notify passengers in real time through AR technology.

[0025] Specifically, the intelligent safety and dynamic optimization module S500 includes: ensuring the safety of the pick-up point, dynamically adjusting the area where the pick-up point is generated, and avoiding generating the pick-up point in dangerous and inaccessible areas; the implementation of the intelligent safety and dynamic optimization module S500 includes: safety assessment and optimization; the safety assessment and optimization dynamically adjusts the generation location of the pick-up point through the graph-based path planning algorithm Dijkstra combined with real-time environmental data to ensure target safety; the real-time environmental data includes pedestrian flow and traffic conditions; at the same time, a risk prediction model is used: risk assessment based on the Bayesian network, predicting potential dangers in advance, and optimizing the generation location of the pick-up point according to potential risks; through the graph-based path planning algorithm Dijkstra combined with real-time environmental data and risk assessment based on the Bayesian network, the generation location of the pick-up point is dynamically adjusted and optimized while considering the path safety of the autonomous driving vehicle to the pick-up point and the risk of the pick-up point.

[0026] The beneficial effects of the present invention are:

[0027] The present invention provides a system for generating pick-up and drop-off points for autonomous driving vehicles based on spatial mapping and environmental reconstruction; the virtual pick-up and drop-off areas generated by the system are generated based on multimodal data fusion, spatial environment mapping and deep Q network, and the positions of the generated virtual pick-up and drop-off points will be closer to the actual positions; the system can perceive and map the surrounding environment in real time, dynamically generate virtual pick-up and drop-off areas, and the generated virtual pick-up and drop-off areas can not only be automatically generated according to environmental changes, but also be visualized through virtual signs and AR display technologies, so as to help passengers and vehicles achieve accurate docking in complex environments; in the process of deep Q network training, the extraction of environmental perception vector X is given. i The corresponding output is the optimal pick-up point decision Y i As long as the number of grids divided into grids is large enough, through X i and the corresponding Y i The model prediction results obtained by deep Q network training will be more accurate; the system can adjust the pick-up point location and generate the corresponding pick-up point route in real time according to the personalized needs of passengers, ensuring driving safety while enhancing the passenger experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the system for generating pick-up and drop-off points for autonomous driving vehicles based on spatial mapping and environmental reconstruction according to the present invention.

[0029] Figure 2 A flow chart is provided for generating a pick-up and drop-off point for an autonomous driving vehicle based on spatial mapping and environmental reconstruction according to the present invention. DETAILED DESCRIPTION

[0030] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given here are only used to illustrate and explain the present invention, and cannot be used to limit the present invention.

[0031] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0032] like Figure 1 As shown, it is a schematic diagram of the autonomous driving vehicle pick-up point generation system based on spatial mapping and environmental reshaping of the present invention, which includes: a spatial perception and mapping module S100; a virtual pick-up area generation module S200; a personalized environmental reshaping module S300; an augmented reality AR and guidance module S400; and an intelligent safety and dynamic optimization module S500.

[0033] The spatial perception and mapping module S100 is used to perceive the surrounding environment through multimodal sensors, including LiDAR, cameras and GPS, generate a three-dimensional spatial model, and identify areas suitable for generating virtual pick-up points; the spatial perception and mapping module includes real-time spatial mapping and dynamic environmental analysis; real-time spatial mapping generates a three-dimensional spatial model through real-time environmental perception, and identifies buildings, streets, pedestrian flow areas and traffic jams through the generated three-dimensional spatial model, providing a basis for the generation of virtual pick-up areas; dynamic environmental analysis will include: combining traffic data, weather information and temporary events to dynamically adjust the three-dimensional spatial model to ensure the safety and accessibility of virtual pick-up points.

[0034] Specifically, the implementation of the spatial perception and mapping module S100 includes: multimodal data fusion, deep learning environment reconstruction and dynamic scene understanding; wherein, the multimodal data fusion uses the extended Kalman filter EKF for multimodal data fusion; the multimodal data is obtained through sensor input, and the sensor input includes: LiDAR, camera and GPS input data; the LiDAR captures high-precision three-dimensional point cloud data of the surrounding environment; the camera obtains information including roads, pedestrians, vehicles and buildings through image processing technology; GPS provides vehicle positioning and motion information; EKF combines the obtained sensor data to obtain the precise position and posture of the target in the environment, and updates it in real time; the target state update formula is:

[0035] S t =F(S t－1 , Z t , U t )+W t

[0036] Among them, St is the target state at the current moment, F is the state transfer matrix, S t－1 is the target state at the previous moment, Z t is the sensor measurement value vector, U t is the control input, W t is the process noise.

[0037] Specifically, deep learning environment reconstruction includes: on the basis of obtaining the precise position and posture information of the target in the environment by EKF, using convolutional neural network CNN combined with graph neural network GNN to perform deep learning to process multimodal data, identify the key features of the target at the corresponding position in the environment, and establish the connection between different targets; including identifying: obstacles, pedestrians and traffic lights, and generating a refined three-dimensional model; the steps of generating a three-dimensional model are: S101, fusing point cloud and image data: fusing the point cloud model fused by EKF and the depth map extracted by CNN to generate a more accurate three-dimensional point cloud; S102, performing three-dimensional reconstruction: generating a three-dimensional geometric model of the environment through a three-dimensional reconstruction algorithm triangular mesh based on the generated three-dimensional point cloud, and the three-dimensional geometric model includes the shape and position of all perceived targets; then, using GNN to model the spatial relationship between different targets and further optimize the model; S103, optimizing the obtained three-dimensional model: modeling the interaction between different targets based on GNN, further optimizing the three-dimensional model, optimizing the relative position and shape of the target, and ensuring the geometric accuracy and semantic consistency of the model; and using GN N optimizes the entire three-dimensional model to ensure the accuracy and integrity of the model; S104, generates a refined three-dimensional model: through steps S101, S102 and S103 combined with the dynamic estimation of EKF, a global refined three-dimensional model is generated through GNN.

[0038] Specifically, dynamic scene understanding includes: detecting and identifying objects in dynamic scenes in real time by integrating YOLOv5, including identifying pedestrians and vehicles in motion; using a time series analysis model based on a temporal convolutional network (TCN) to predict traffic flow and pedestrian flow dynamics; the steps of dynamic scene understanding are: S201, first, processing each video frame in the dynamic scene by YOLOv5, identifying the target and the position of the target in each video frame, wherein the target includes pedestrians, vehicles and obstacles; S202, arranging the target detection results output by YOLOv5 in chronological order and using them as inputs of a TCN model, and training the TCN model to understand the behavior of the target in time sequence; S203, the trained TCN model further optimizes the tracking of the object by capturing the motion pattern of the object; at the same time, while identifying the motion trajectory of the target, the motion state of the target is predicted, and the system makes different decisions for the dynamic scene according to the different motion states and motion patterns of the target, wherein the different decisions include: honking, stopping and decelerating.

[0039] The virtual pick-up area generation module S200 is used to dynamically generate a virtual pick-up area based on the spatial perception and mapping module S100; the generated area not only includes establishing suitable pick-up points in areas with convenient transportation, but also includes temporarily creating pick-up points in urban congested areas and activity venues, and adjusting according to passenger needs and environmental changes; the virtual pick-up area generation module S200 includes: virtual pick-up point creation and pick-up area adjustment; the virtual pick-up point creation is based on the spatial mapping results generated by the spatial perception and mapping module S100, and when there are no fixed pick-up points and the existing pick-up points are unreachable, a virtual pick-up point suitable for pick-up is generated; the pick-up area adjustment is used when the environment changes, the system automatically adjusts the pick-up area, generates a new virtual pick-up point, and notifies passengers and vehicles in real time.

[0040] Specifically, the virtual pick-up area generation module S200 dynamically generates an optimal virtual pick-up area according to real-time spatial perception data; the implementation of the virtual pick-up area generation module S200 includes: optimal pick-up point generation, multi-objective optimization and dynamic adjustment mechanism; the optimal pick-up point generation includes: dynamically generating the pick-up area using the deep Q network in the deep reinforcement learning technology; the steps of dynamically generating the pick-up area by the deep Q network are: S301, extracting the historical multi-modal fused environmental perception data to form an environmental perception vector X i , i represents the i-th group of environmental perception vectors; S302, extracting the decision Y of the optimal pick-up point corresponding to the environmental perception vector i ; S303, X i As the input of the deep Q network, the corresponding Y i The deep Q network is trained as the output of the deep Q network. During the training process, the model continuously optimizes the location of the pick-up point according to passenger needs, environmental restrictions and safety constraints to obtain a trained deep Q network model, wherein the trained deep Q network model refers to a difference between the model predicted Q value and the target Q value that meets a preset condition; S304, extracting multimodal data information of the autonomous driving vehicle in real time, processing the multimodal data and inputting it into the trained deep Q network model, and judging the optimal pick-up point according to the output result; the decision Y of the optimal pick-up point corresponding to the environmental perception vector is extracted iThe steps are as follows: S401, dividing the entire image area corresponding to the environment perception vector into grids with horizontal and vertical lines; S402, determining whether each grid meets the pick-up and drop-off requirements according to the environment perception vector, if the corresponding grid includes pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are not met; if the corresponding grid does not include pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are met; S403, extracting an area composed of continuous grids that meet the pick-up and drop-off requirements in the entire image area as an alternative pick-up and drop-off area, and judging whether the shape and size of the autonomous driving vehicle can be placed in the alternative pick-up and drop-off area according to the shape and size of the autonomous driving vehicle; the area composed of continuous grids that meet the pick-up and drop-off requirements means that each grid in the alternative pick-up and drop-off area meets the pick-up and drop-off requirements, and each grid that meets the pick-up and drop-off requirements meets the pick-up and drop-off requirements. The grid to be delivered is at least continuous with another grid that meets the pick-up requirements, and the grids that meet the pick-up requirements in the candidate pick-up area formed must be filled with grids that meet the pick-up requirements; S404, select the candidate pick-up area whose shape and size can be placed in the autonomous driving vehicle as the feasible pick-up area, and determine the optimal pick-up point according to the shape regularity, area, safety and whether the vehicle is allowed to enter the feasible pick-up area; set all grid values ​​in the candidate pick-up area corresponding to the optimal pick-up point to 1, set all values ​​of other grids in the entire image area to 0, set the values ​​set for the horizontal rows of the grids in the entire image area as the values ​​of the horizontal rows of the matrix, and set the values ​​set for the vertical rows of the grids in the entire image area as the values ​​of the vertical rows of the matrix, to form the decision matrix Y of the optimal pick-up point in the entire image area i .

[0041] Specifically, the multi-objective optimization includes: using the Pareto optimal solution multi-objective optimization algorithm to weigh the safety, passenger convenience and arrival time of different feasible pick-up areas to ensure the optimal location of the pick-up point; the dynamic adjustment mechanism includes: when the environment changes, the system updates the location of the pick-up point in real time through strategy adjustment based on Markov decision MDP.

[0042] The personalized environment reshaping module S300 reshapes the virtual pick-up area according to the personalized needs of each passenger, so that the reshaped virtual pick-up area meets the special requirements of the passengers; the personalized environment reshaping module S300 includes: personalized pick-up demand matching and dynamic environment reshaping; personalized pick-up demand matching is used to give priority to matching the personalized needs of passengers when generating virtual pick-up areas based on the historical behavior and preferences of passengers; dynamic environment reshaping is used to make personalized adjustments to the virtual pick-up point based on the passenger input needs and current environmental conditions to ensure that the vehicle can meet the passengers at the optimal location.

[0043] Specifically, the personalized environment reshaping module S300 includes: generating and adjusting virtual pick-up points according to the personalized needs of passengers; the implementation of the personalized environment reshaping module S300 includes: personalized demand matching and personalized environment reshaping; wherein, personalized demand matching includes: analyzing the historical behavior and preferences of passengers based on the recommendation system algorithm of collaborative filtering, and generating virtual pick-up areas that meet the needs of passengers; the recommendation system algorithm based on collaborative filtering extracts implicit passenger behavior patterns from massive passenger data through matrix decomposition technology, and generates personalized virtual pick-up points under passenger needs by applying conditional generative adversarial networks CGAN, and the conditional input of the conditional generative adversarial networks CGAN includes passenger preferences and current environmental restrictions; the personalized environment reshaping includes: dynamically adjusting the location of the virtual pick-up point through the Bayesian optimization algorithm to ensure that the pick-up point meets the personalized needs of passengers, especially in special locations including building entrances, parking lots and away from noise areas; the recommendation system algorithm of collaborative filtering performs score prediction based on the needs, historical behaviors and preferences of passengers, and predicts the pick-up points that meet the needs of passengers. The collaborative filtering score prediction formula is:

[0044]

[0045] in, represents the predicted score, which represents the score of passenger u for pick-up point i; μ is the global bias, b u and b i denote the offsets of the passenger and the pick-up point, respectively, q i and p u are the hidden vectors of the pick-up point and the passenger respectively, and T is q i The transpose of .

[0046] The CGAN generation model is: P(x|c)=G(z,c), where z is random noise, c is the passenger's personalized demand, x is the pick-up point location, G is the model generator, and P is the probability distribution of the pick-up point location x under given conditions c.

[0047] Among them, the augmented reality AR and guidance module S400 provides visual guidance for passengers through augmented reality AR technology, helping passengers to quickly and accurately find the virtual pick-up area; the augmented reality AR and guidance module S400 includes: AR signs and guidance and real-time AR feedback; the AR signs and guidance generate virtual pick-up signs through the passenger's mobile device, and guide the passenger from the current location to the virtual pick-up sign through the AR interface; the real-time AR feedback means that when the passenger approaches the pick-up area, the system will provide real-time feedback through the AR interface to guide the passenger to accurately locate the vehicle.

[0048] Specifically, the augmented reality AR and guidance module S400 includes: providing visual guidance for passengers through AR technology to help them reach the pick-up area quickly and accurately; the implementation of the augmented reality AR and guidance module S400 includes: AR visual navigation and real-time path guidance; wherein, AR visual navigation includes: using SLAM technology to implement AR navigation on the passenger's mobile device; the SLAM algorithm scans the environment in real time through the device camera, builds a map of the surrounding environment, and locates the position of the passenger's mobile device; combined with the deep learning visual feature extraction network ResNet50 to detect surrounding objects, and generate virtual pick-up point signs based on the identified surrounding key objects and areas; through the AR system, the surrounding environment is projected in real time and the path to the pick-up point is planned. The passenger views the generated virtual pick-up point sign and the generated path plan through the mobile device, and preliminarily generates the path from the passenger to the pick-up point; the real-time path guidance includes: based on the path planning algorithm Dijkstra algorithm combined with the obstacle information in the AR system, dynamically adjust the path from the passenger to the pick-up point; at the same time, the system will display dynamic signs through the AR interface when the passenger approaches the pick-up point, and provide accurate navigation instructions; the SLAM navigation equation is:

[0049] x t =f(x t－1 ,u t )+w t

[0050] Among them, x t is the passenger’s current position, x t－1 The passenger's position at the last moment, f(x t－1 ,u t ) is the state transfer function, u t is the control input, w t is the process noise.

[0051] Among them, the intelligent safety and dynamic optimization module S500 is used to ensure the safety and dynamic optimization of autonomous driving vehicles while the system generates a virtual pick-up area, and to avoid generating pick-up points in dangerous and inaccessible areas; the intelligent safety and dynamic optimization module S500 includes: safety priority generation and dynamic adjustment and feedback; safety priority generation means that when the system generates a virtual pick-up area, it combines: pedestrian flow data, traffic conditions and temporary obstacle information to ensure the safety of the pick-up area and avoid generating pick-up points on highways and congested areas; dynamic adjustment and feedback are used when the environment changes, the system will automatically adjust the virtual pick-up area, and notify passengers in real time through AR technology.

[0052] Specifically, the intelligent safety and dynamic optimization module S500 includes: ensuring the safety of the pick-up point, dynamically adjusting the area where the pick-up point is generated, and avoiding generating the pick-up point in dangerous and inaccessible areas; the implementation of the intelligent safety and dynamic optimization module S500 includes: safety assessment and optimization; the safety assessment and optimization dynamically adjusts the generation location of the pick-up point through the graph-based path planning algorithm Dijkstra combined with real-time environmental data to ensure target safety; the real-time environmental data includes pedestrian flow and traffic conditions; at the same time, a risk prediction model is used: risk assessment based on the Bayesian network, predicting potential dangers in advance, and optimizing the generation location of the pick-up point according to potential risks; through the graph-based path planning algorithm Dijkstra combined with real-time environmental data and risk assessment based on the Bayesian network, the generation location of the pick-up point is dynamically adjusted and optimized while considering the path safety of the autonomous driving vehicle to the pick-up point and the risk of the pick-up point.

[0053] like Figure 2 A flowchart for generating a pick-up point for an autonomous driving vehicle based on spatial mapping and environmental reconstruction of the present invention is provided, and the flowchart includes: the implementation of the system is provided with a global controller, and the global controller includes all modules of the system and can realize the functions of the corresponding modules; first, the global controller extracts the three-dimensional environmental information and dynamically updated environmental changes in the collected information through the spatial perception and mapping module S100; the extracted information includes: data collected by LiDAR, camera, GPS and various sensors; then, the dynamically updated environmental changes are transmitted to the intelligent safety and dynamic optimization module S500; the intelligent safety and dynamic optimization module S500 and the provided three-dimensional environmental data are used to analyze the safety of the pick-up point, and the analysis results are transmitted to the virtual pick-up area generation module Block S200; the virtual pick-up area generation module S200 generates an initial pick-up point, and combines the safety adjustment information fed back by the intelligent safety and dynamic optimization module S500, and transmits it to the personalized environment reshaping module S300. The personalized environment reshaping module S300 generates a personalized pick-up point through personalized adjustments to the pick-up point; the personalized reshaped environment and the personalized adjusted pick-up point are transmitted to the augmented reality AR and guidance module S400 together, and the AR display navigation of the augmented reality AR and guidance module S400 is used to navigate the passengers to help them approach the pick-up point; when the passengers approach the pick-up point, the information is fed back to the augmented reality AR and guidance module S400 in real time, so as to realize real-time interaction between the passengers and the system, help the passengers quickly reach the pick-up point, and enhance the passenger experience.

Claims

1. A system for generating pick-up and drop-off points for autonomous vehicles based on spatial mapping and environmental reconstruction, characterized in that: The system comprises: Spatial perception and mapping module S100; virtual pick-up area generation module S200; personalized environment reshaping module S300; augmented reality AR and guidance module S400; intelligent safety and dynamic optimization module S500; The spatial perception and mapping module S100 is used to perceive the surrounding environment through multimodal sensors, including LiDAR, camera and GPS, generate a three-dimensional spatial model, and identify areas suitable for generating virtual pick-up points; the spatial perception and mapping module includes real-time spatial mapping and dynamic environmental analysis; the real-time spatial mapping generates a three-dimensional spatial model through real-time environmental perception, identifies buildings, streets, pedestrian flow areas and traffic jams through the generated three-dimensional spatial model, and provides a basis for the generation of virtual pick-up areas; the dynamic environmental analysis will include: combining traffic data, weather information and temporary events to dynamically adjust the three-dimensional spatial model to ensure the safety and accessibility of the virtual pick-up points; The virtual pick-up area generation module S200 is used to dynamically generate a virtual pick-up area based on the space perception and mapping module S100; the generated area not only includes establishing suitable pick-up points in areas with convenient transportation, but also includes temporarily creating pick-up points in urban congested areas and activity venues, and adjusting according to passenger needs and environmental changes; the virtual pick-up area generation module S200 includes: virtual pick-up point creation and pick-up area adjustment; the virtual pick-up point creation is based on the spatial mapping results generated by the space perception and mapping module S100, and generates a virtual pick-up point suitable for pick-up when there is no fixed pick-up point and the existing pick-up point is unreachable; the pick-up area adjustment is used for when the environment changes, the system automatically adjusts the pick-up area, generates a new virtual pick-up point, and notifies passengers and vehicles in real time; The personalized environment reshaping module S300 reshapes the virtual pick-up area according to the personalized needs of each passenger, so that the reshaped virtual pick-up area meets the special requirements of the passenger; the personalized environment reshaping module S300 includes: personalized pick-up demand matching and dynamic environment reshaping; personalized pick-up demand matching is used to give priority to matching the personalized needs of the passenger when generating the virtual pick-up area in combination with the historical behavior and preferences of the passenger; dynamic environment reshaping is used to make personalized adjustments to the virtual pick-up point according to the passenger input needs and current environmental conditions, so as to ensure that the vehicle can meet the passenger at the optimal location; The augmented reality AR and guidance module S400 provides visual guidance for passengers through augmented reality AR technology, helping passengers to quickly and accurately find the virtual pick-up area; the augmented reality AR and guidance module S400 includes: AR signs and guidance and real-time AR feedback; the AR signs and guidance generate virtual pick-up signs through the passenger's mobile device, and guide the passenger from the current location to the virtual pick-up sign through the AR interface; the real-time AR feedback means that when the passenger approaches the pick-up area, the system will provide real-time feedback through the AR interface to guide the passenger to accurately locate the vehicle; The intelligent safety and dynamic optimization module S500 is used to ensure the safety and dynamic optimization of the autonomous driving vehicle while the system generates a virtual pick-up area, and avoid generating pick-up points in dangerous and inaccessible areas; the intelligent safety and dynamic optimization module S500 includes: safety priority generation and dynamic adjustment and feedback; safety priority generation means that when the system generates a virtual pick-up area, it combines: pedestrian flow data, traffic conditions and temporary obstacle information to ensure the safety of the pick-up area and avoid generating pick-up points on highways and congested areas; dynamic adjustment and feedback is used when the environment changes, the system automatically adjusts the virtual pick-up area, and notifies passengers in real time through AR technology.

2. The autonomous driving vehicle pick-up and drop-off point generation system based on spatial mapping and environment reconstruction according to claim 1, characterized in that: The implementation of the spatial perception and mapping module S100 includes: multimodal data fusion, deep learning environment reconstruction and dynamic scene understanding; wherein, the multimodal data fusion uses the extended Kalman filter EKF for multimodal data fusion; the multimodal data is obtained through sensor input, and the sensor input includes: LiDAR, camera and GPS input data; the LiDAR captures high-precision three-dimensional point cloud data of the surrounding environment; the camera obtains information including roads, pedestrians, vehicles and buildings through image processing technology; the GPS provides vehicle positioning and motion information; the EKF combines the obtained sensor data to obtain the precise position and posture of the target in the environment, and updates it in real time; the target state update formula is: S t =F(S t－1 ,Z t ,U t )+W t Among them, S t is the target state at the current moment, F is the state transfer matrix, S t－1 is the target state at the previous moment, Z t is the sensor measurement value vector, U t is the control input, W t is the process noise; The deep learning environment reconstruction includes: based on the EKF to obtain the precise position and posture information of the target in the environment, using the convolutional neural network CNN combined with the graph neural network GNN to perform deep learning to process multimodal data, identify the key features of the corresponding position targets in the environment, and establish connections between different targets; including identifying: obstacles, pedestrians and traffic lights, and generating a refined three-dimensional model; the steps of generating the three-dimensional model are: S101, fusing the point cloud and image data: fusing the point cloud model fused by EKF and the depth map extracted by CNN to generate a more accurate three-dimensional point cloud; S102, performing three-dimensional reconstruction: generating the environment through a triangular mesh of a three-dimensional reconstruction algorithm based on the generated three-dimensional point cloud A three-dimensional geometric model is obtained, wherein the three-dimensional geometric model includes the shapes and positions of all perceived targets; subsequently, the spatial relationship between different targets is modeled using GNN to further optimize the model; S103, the obtained three-dimensional model is optimized: based on GNN, the interaction between different targets is modeled, the three-dimensional model is further optimized, the relative position and shape of the targets are optimized, and the geometric accuracy and semantic consistency of the model are ensured; and the entire three-dimensional model is optimized using GNN to ensure the accuracy and integrity of the model; S104, a refined three-dimensional model is generated: through steps S101, S102 and S103 combined with the dynamic estimation of EKF, a global refined three-dimensional model is generated by GNN; The dynamic scene understanding includes: detecting and identifying objects in dynamic scenes in real time by integrating YOLOv5, including identifying pedestrians and vehicles in motion; using a time series analysis model based on a temporal convolutional network (TCN) to predict traffic flow and pedestrian flow dynamics; the steps of dynamic scene understanding are: S201, first, processing each video frame in the dynamic scene by YOLOv5, identifying the target and the position of the target in each video frame, wherein the target includes pedestrians, vehicles and obstacles; S202, arranging the target detection results output by YOLOv5 in chronological order and using them as inputs of a TCN model, and training the TCN model to understand the behavior of the target in time sequence; S203, the trained TCN model further optimizes the tracking of the object by capturing the motion pattern of the object; at the same time, while identifying the motion trajectory of the target, predicting the motion state of the target, the system makes different decisions for the dynamic scene according to the different motion states and motion patterns of the target, wherein the different decisions include: honking, stopping and decelerating.

3. The autonomous driving vehicle pick-up and drop-off point generation system based on spatial mapping and environment reconstruction according to claim 1, characterized in that: The virtual pick-up area generation module S200 includes: dynamically generating an optimal virtual pick-up area according to real-time spatial perception data; the implementation of the virtual pick-up area generation module includes: optimal pick-up point generation, multi-objective optimization and dynamic adjustment mechanism; wherein, the optimal pick-up point generation includes: dynamically generating the pick-up area using a deep Q network in deep reinforcement learning technology; the steps of dynamically generating the pick-up area using the deep Q network are: S301, extracting historical multi-modal fused environmental perception data to form an environmental perception vector X i , i represents the i-th group of environmental perception vectors; S302, extracting the decision Y of the optimal pick-up point corresponding to the environmental perception vector i ; S303, X i As the input of the deep Q network, the corresponding Y i The deep Q network is trained as the output of the deep Q network. During the training process, the model continuously optimizes the location of the pick-up point according to passenger needs, environmental restrictions and safety constraints to obtain a trained deep Q network model, wherein the trained deep Q network model refers to a difference between the model predicted Q value and the target Q value that meets a preset condition; S304, extracting multimodal data information of the autonomous driving vehicle in real time, processing the multimodal data and inputting it into the trained deep Q network model, and judging the optimal pick-up point according to the output result; the decision Y of the optimal pick-up point corresponding to the environmental perception vector is extracted i The steps are as follows: S401, dividing the entire image area corresponding to the environment perception vector into grids with horizontal and vertical lines; S402, determining whether each grid meets the pick-up and drop-off requirements according to the environment perception vector, if the corresponding grid includes pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are not met; if the corresponding grid does not include pedestrians, vehicles and obstacles, it is determined that the pick-up and drop-off requirements are met; S403, extracting an area composed of continuous grids that meet the pick-up and drop-off requirements in the entire image area as an alternative pick-up and drop-off area, and judging whether the shape and size of the autonomous driving vehicle can be placed in the alternative pick-up and drop-off area according to the shape and size of the autonomous driving vehicle; the area composed of continuous grids that meet the pick-up and drop-off requirements means that each grid in the alternative pick-up and drop-off area meets the pick-up and drop-off requirements, and each grid that meets the pick-up and drop-off requirements meets the pick-up and drop-off requirements. The grid to be delivered is at least continuous with another grid that meets the pick-up requirements, and the grids that meet the pick-up requirements in the candidate pick-up area formed must be filled with grids that meet the pick-up requirements; S404, select the candidate pick-up area whose shape and size can be placed in the autonomous driving vehicle as the feasible pick-up area, and determine the optimal pick-up point according to the shape regularity, area, safety and whether the vehicle is allowed to enter the feasible pick-up area; set all grid values ​​in the candidate pick-up area corresponding to the optimal pick-up point to 1, set all values ​​of other grids in the entire image area to 0, set the values ​​set for the horizontal rows of the grids in the entire image area as the values ​​of the horizontal rows of the matrix, and set the values ​​set for the vertical rows of the grids in the entire image area as the values ​​of the vertical rows of the matrix, to form the decision matrix Y of the optimal pick-up point in the entire image area i ; The multi-objective optimization includes: using the multi-objective optimization algorithm of Pareto optimal solution to weigh the safety, passenger convenience and arrival time of different feasible pick-up areas to ensure the optimal location of the pick-up point; the dynamic adjustment mechanism includes: when the environment changes, the system updates the location of the pick-up point in real time through strategy adjustment based on Markov decision MDP.

4. The autonomous driving vehicle pick-up and drop-off point generation system based on spatial mapping and environment reconstruction according to claim 1, characterized in that: The personalized environment reshaping module S300 includes: generating and adjusting virtual pick-up points according to the personalized needs of passengers; the implementation of the personalized environment reshaping module includes: personalized demand matching and personalized environment reshaping; wherein, personalized demand matching includes: analyzing the historical behavior and preferences of passengers based on the recommendation system algorithm of collaborative filtering, and generating virtual pick-up areas that meet the needs of passengers; the recommendation system algorithm based on collaborative filtering extracts implicit passenger behavior patterns from massive passenger data through matrix decomposition technology, and generates personalized virtual pick-up points under passenger needs by applying conditional generative adversarial network CGAN, and the conditional input of the conditional generative adversarial network CGAN includes passenger preferences and current environmental restrictions; the personalized environment reshaping includes: dynamically adjusting the location of the virtual pick-up point through the Bayesian optimization algorithm to ensure that the pick-up point meets the personalized needs of passengers, especially in special locations including building entrances, parking lots and away from noise areas; the recommendation system algorithm of collaborative filtering predicts scores based on the needs, historical behaviors and preferences of passengers, and predicts pick-up points that meet the needs of passengers. The collaborative filtering score prediction formula is: in, represents the predicted score, which represents the score of passenger u for pick-up point i; μ is the global bias, b u and b i denote the offsets of the passenger and the pick-up point, respectively, q i and p u are the latent vectors of the pick-up point and the passenger respectively, and T is q i The transpose of The CGAN generation model is: P(xc)=G(z,c), where z is random noise, c is the passenger's personalized demand, x is the pick-up point location, G is the model generator, and P is the probability distribution of the pick-up point location x under given conditions c.

5. The autonomous driving vehicle pick-up and drop-off point generation system based on spatial mapping and environment reconstruction according to claim 1, characterized in that: The augmented reality AR and guidance module S400 includes: providing visual guidance for passengers through AR technology to help them reach the pick-up area quickly and accurately; the implementation of the augmented reality AR and guidance module S400 includes: AR visual navigation and real-time path guidance; wherein, AR visual navigation includes: using SLAM technology to implement AR navigation on the passenger's mobile device; the SLAM algorithm scans the environment in real time through the device camera, builds a map of the surrounding environment, and locates the position of the passenger's mobile device; combines the deep learning visual feature extraction network ResNet50 to detect surrounding objects, and generates a virtual pick-up point sign based on the identified surrounding key objects and areas; through the AR system, the surrounding environment is projected in real time and the path to the pick-up point is planned, and the passenger views the generated virtual pick-up point sign and the generated path plan through the mobile device, and preliminarily generates the path from the passenger to the pick-up point; the real-time path guidance includes: based on the path planning algorithm Dijkstra algorithm combined with the obstacle information in the AR system, dynamically adjusting the path from the passenger to the pick-up point; at the same time, the system will display dynamic signs through the AR interface when the passenger approaches the pick-up point, and provide accurate navigation instructions; the SLAM navigation equation is: x t =f(x t－1 ,u t )+w t Among them, x t is the passenger’s current position, x t－1 The passenger's position at the last moment, f(x t－1 ,u t ) is the state transfer function, u t is the control input, w t is the process noise.

6. The autonomous driving vehicle pick-up and drop-off point generation system based on spatial mapping and environment reconstruction according to claim 1, characterized in that: The intelligent safety and dynamic optimization module S500 includes: ensuring the safety of the pick-up point, dynamically adjusting the area where the pick-up point is generated, and avoiding generating the pick-up point in dangerous and inaccessible areas; the implementation of the intelligent safety and dynamic optimization module S500 includes: safety assessment and optimization; the safety assessment and optimization dynamically adjusts the generation location of the pick-up point through the graph-based path planning algorithm Dijkstra combined with real-time environmental data to ensure target safety; the real-time environmental data includes pedestrian flow and traffic conditions; at the same time, a risk prediction model is used: risk assessment based on the Bayesian network, predicting potential dangers in advance, and optimizing the generation location of the pick-up point according to potential risks; through the graph-based path planning algorithm Dijkstra combined with real-time environmental data and risk assessment based on the Bayesian network, the generation location of the pick-up point is dynamically adjusted and optimized while considering the path safety of the autonomous driving vehicle to the pick-up point and the risk of the pick-up point.

Images