Attention mechanism-based man-like parking trajectory planning method, system and equipment, and medium

Abstract

The invention provides a human-like parking track planning method, system and device based on an attention mechanism and a medium, and relates to the field of automatic driving and reinforcement learning, and the method comprises the steps: S1, constructing a training environment for simulating a parking scene; s2, constructing an attention network by adopting a mixed attention mechanism according to training information in the parking scene; s3, training the reinforcement learning model, and adding a human preference-based guide mechanism into the model; s4, initializing an Actor network and a Critic network of the learning model, and training the learning model through the constructed simulated parking scene; and S5, based on the input training information in the parking scene, outputting a steering wheel rotation angle and acceleration instruction, obtaining an automatic parking track plan of the current automatic driving vehicle, and completing automatic parking. According to the invention, the efficiency and safety of automatic parking are improved, safer and more efficient automatic parking track planning is realized, and the driving track is more in line with human driving habits.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and reinforcement learning technologies, specifically to a human-like parking trajectory planning method, system, device, and medium based on an attention mechanism. Background Technology

[0002] In recent years, there has been increasing interest in safe and efficient planning in unstructured scenarios, with autonomous valet parking being a typical example. Compared to structured scenarios such as highways, autonomous valet parking requires higher precision in confined spaces with a mix of static and dynamic obstacles.

[0003] Current research methods for autonomous valet parking mainly include heuristic search methods based on motion primitives, optimization methods based on multi-objective constraints, and learning-based methods. Search-based methods find the global optimum under the constraints of motion primitives, but struggle to handle complex environments. Optimization-based methods follow kinematic constraints, but are computationally complex and sensitive to initial conditions. Furthermore, both of these methods are currently only applicable to static scenarios and lack adaptability to dynamic environments. Learning-based methods mainly include imitation learning and reinforcement learning. Imitation learning leverages the powerful model representation capabilities of neural networks to learn efficient driving strategies from high-quality and large-scale expert datasets, but struggles to surpass expert demonstrations. Reinforcement learning learns optimal strategies through interaction with the environment, enabling automatic strategy adjustments to adapt to different environments and task requirements.

[0004] Current research on automated parking based on deep reinforcement learning mainly involves observing surrounding environmental information and directly mapping it to accelerator and steering wheel angles to control the vehicle's movement to the target parking space. Only a few works have achieved some expected results, and most are only applied to simple scenarios with static obstacles. They do not consider the final parking posture, nor do they guide the parking process, which is inconsistent with human driving habits and suffers from drawbacks such as long training times and low sampling efficiency. In unstructured and highly dynamic parking scenarios, it is necessary to consider not only heterogeneous information such as dynamic obstacles, the irregular distribution of static obstacles, and the target parking space, but also to simulate a human-like driving trajectory during the parking process to ensure the final parking posture. Summary of the Invention

[0005] In view of this, embodiments of this application provide a human-like parking trajectory planning method, system, device, and medium based on an attention mechanism. First, a training environment simulating parking scenarios is built, capable of accepting actions from reinforcement learning input and returning an updated environment state. Then, a hybrid attention mechanism is designed to extract information features of dynamic and static obstacles in complex parking scenarios, effectively capturing the interaction between heterogeneous traffic information, including vehicle state, other obstacles, and target point information. Finally, a guidance mechanism based on human preferences is designed to enable the agent to learn effective intermediate parking strategies, accelerating the training process and making the driving trajectory more consistent with human driving habits.

[0006] This application provides the following technical solution: a human-like parking trajectory planning method based on an attention mechanism, comprising the following steps: S1. Construct a training environment for simulating parking scenarios, and represent the training information in the parking scenarios using vectors. The training information includes the current position, orientation and speed of the autonomous vehicle, the position, orientation and speed of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. S2. Based on the training information in the parking scenario, an attention network is constructed using a hybrid attention mechanism, so that the attention network outputs a corresponding feature vector based on the hybrid dynamic and static information of the input training information; S3. Train the reinforcement learning model and add a guidance mechanism based on human preferences to the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. S4. Initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, and optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the hybrid type reward function. S5. Based on the training information in the input parking scenario, the trained learning model outputs steering wheel angle and acceleration commands to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

[0007] According to one embodiment of this application, the constructed simulated parking scenario is as follows: The state information in the scene includes the current autonomous vehicle, the target parking space, other moving and static vehicles besides the current autonomous vehicle, and static obstacles; each state information is represented using a 6-dimensional vector data. , representing position, speed, and orientation angle information respectively, and distinguishing between static and dynamic vehicles by speed, where the speed parameter in the target parking space vector is 0.

[0008] According to one embodiment of this application, an attention network is constructed using a hybrid attention mechanism, including: The attention network encodes the current vehicle state and the surrounding static and dynamic vehicle states, extracts information features of dynamic and static obstacles, captures the interaction between heterogeneous traffic information, and constructs the attention network based on the training information in the parking scenario.

[0009] According to one embodiment of this application, the human preference-based guidance mechanism adaptively generates intermediate guidance positions based on human preferences through the initial position of the current vehicle and the target parking space information. An arrival reward mechanism is added to each intermediate guidance position. Based on the current vehicle state, the position weight function is used to determine whether the agent has reached the intermediate guidance position. If so, the arrival reward mechanism is executed to guide the behavior of the model agent.

[0010] According to one embodiment of this application, the DDPG algorithm network structure includes an Actor network and a Critic network, wherein the outputs of the Actor network and the Critic network are different, while the rest of the structure is the same. The inputs to the Actor network and the Critic network are vector information of the autonomous vehicle, the target parking space, other dynamic and static vehicles and static obstacles besides the current autonomous vehicle, and the intermediate guide position. The information of the autonomous vehicle and other dynamic and static vehicles and static obstacles is input into the attention network, which outputs a corresponding feature vector. This feature vector is then concatenated with the other state information. The Actor network and the Critic network are connected and output through two fully connected layers, and are activated by the Tanh() function. The Actor network directly outputs the steering wheel angle and acceleration, and the last layer of the Critic network directly outputs a value.

[0011] This application also provides a human-like parking trajectory planning system based on an attention mechanism, including: The parking scenario construction and preprocessing module is used to construct a training environment for simulating parking scenarios. The training information in the parking scenario is represented by vectors. The training information includes the current autonomous vehicle's position, orientation and speed, the positions, orientations and speeds of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. The hybrid attention mechanism module is used to construct an attention network based on the training information in the parking scenario, so that the attention network outputs a corresponding feature vector based on the mixed dynamic and static information of the input training information. The intermediate region guidance module is used to train the reinforcement learning model. It incorporates a guidance mechanism based on human preferences into the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. The model training module is used to initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, and optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the hybrid type reward function. The parking trajectory output module is used to output steering wheel angle and acceleration commands based on the training information in the input parking scenario and the trained learning model, so as to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

[0012] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described humanoid parking trajectory planning method based on an attention mechanism.

[0013] This application also provides a computer-readable storage medium storing a computer program that executes the above-described attention-based humanoid parking trajectory planning method.

[0014] Compared with existing technologies, the beneficial effects achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: The novel reinforcement learning algorithm proposed in the embodiments of this invention realizes a human-like trajectory planning method in parking environments, which can plan a parking trajectory that conforms to human driving habits in complex scenarios with mixed static and dynamic obstacles. The design of random scene generation enables the reinforcement learning model to handle different complex scenarios, enhancing the robustness and generalization of the model; the design of the attention mechanism and the implementation of the attention network can effectively capture the interaction of heterogeneous traffic information by outputting feature vectors, improving sample efficiency and model training speed; the implementation of the adaptive guidance region mechanism enables the agent to learn human-like operation strategies, improving the efficiency and safety of automatic parking, and also accelerating model training; finally, compared with PPO and DDPG algorithms, the method of this invention can successfully learn human-preferred parking strategies in randomly changing scenarios and can continuously optimize the strategy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a human-like parking reinforcement learning trajectory planning method with an attention mechanism according to an embodiment of the present invention. Figure 2 This is an overall framework diagram of a human-like parking reinforcement learning trajectory planning method with an attention mechanism according to an embodiment of the present invention; Figure 3 A schematic diagram of a parking environment constructed for an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the interaction between the parking environment and the DDPG intelligent agent according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the state information of an autonomous vehicle according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the intermediate guidance region mechanism in an embodiment of the present invention; Figure 7 This is a schematic diagram of the overall network structure of DDPG according to an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating how the reward changes with the number of training rounds, regardless of whether a hybrid attention mechanism is used, in an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating the test results of models PPO (a), DDPG (b), unrestricted target position (c), restricted target position (d), restricted target position, and model with intermediate guiding region (e) in embodiments of the present invention. Detailed Implementation

[0017] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] like Figure 1 As shown, this embodiment of the invention provides a human-like parking trajectory planning method based on an attention mechanism, including the following steps: S1. Construct a training environment for simulating parking scenarios, and represent the training information in the parking scenarios using vectors. The training information includes the current position, orientation and speed of the autonomous vehicle, the position, orientation and speed of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. S2. Based on the training information in the parking scenario, an attention network is constructed using a hybrid attention mechanism, so that the attention network outputs a corresponding feature vector based on the hybrid dynamic and static information of the input training information; S3. Train the reinforcement learning model and add a guidance mechanism based on human preferences to the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. S4. Initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, and optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the hybrid type reward function. S5. Based on the training information in the input parking scenario, the trained learning model outputs steering wheel angle and acceleration commands to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

[0020] In some embodiments of the present invention, the constructed simulated parking scenario is as follows: the state information in the scenario includes the current autonomous driving vehicle, the target parking space, dynamic and static vehicles other than the current autonomous driving vehicle, and static obstacles; wherein, each state information is represented by a 6-dimensional vector data. , representing position, speed, and orientation angle information respectively, and distinguishing between static and dynamic vehicles by speed, where the speed parameter in the target parking space vector is 0.

[0021] In some embodiments of the present invention, an attention network is constructed using a hybrid attention mechanism, including: encoding the current vehicle state and the surrounding static and dynamic vehicle states using the attention network, extracting information features of dynamic and static obstacles, capturing the interaction between heterogeneous traffic information, and constructing the attention network based on the training information in the parking scenario.

[0022] The attention mechanism comprises three functions, Q, K, and V. An attention score is calculated using query Q and key K, and a normalized score is applied to value V. Specifically, the implementation focuses on the impact of other vehicles' information on the vehicle itself. Through an attention network, the vehicle's state and the states of surrounding static and dynamic vehicles are feature-encoded, enabling the extraction of information features from static and dynamic obstacles and effectively capturing the interactions between heterogeneous traffic information. In some embodiments of the present invention, the human preference-based guidance mechanism adaptively generates intermediate guidance positions based on human preferences through the initial position of the current vehicle and the target parking space information. An arrival reward mechanism is added to each intermediate guidance position. Based on the current vehicle state, the position weight function is used to determine whether the agent has reached the intermediate guidance position. If so, the arrival reward mechanism is executed to guide the behavior of the model agent.

[0023] In some embodiments of the present invention, the DDPG algorithm network structure includes an Actor network and a Critic network. The outputs of the Actor network and the Critic network are different, but the rest of the structure is the same. The inputs of the Actor network and the Critic network are vector information of the autonomous vehicle, the target parking space, other dynamic and static vehicles and static obstacles besides the current autonomous vehicle, and the intermediate guide position. The information of the autonomous vehicle and other dynamic and static vehicles and static obstacles is input into the attention network, which outputs a corresponding feature vector. This feature vector is then concatenated with the other state information. The Actor network and the Critic network are connected and output through two fully connected layers, and are activated by the Tanh() function. The Actor network directly outputs the steering wheel angle and acceleration, and the last layer of the Critic network directly outputs a value.

[0024] Figure 2This invention presents the overall framework for a human-like parking trajectory planning method based on reinforcement learning, including parking environment construction, feature processing, adaptive guidance region, attention model, reward design, and reinforcement learning training modules. This invention is applicable to highly complex unstructured parking scenarios with mixed static and dynamic conditions. It acquires state information from randomly generated parking scenarios, including information about the vehicle itself, other vehicles, the target parking space, and obstacles. This information is then processed to suit the input of the neural network. Furthermore, a guidance region is adaptively added based on the parking scenario and incorporated into the feature information processing. The feature information, after being encoded by the attention model, is input into the reinforcement learning network, where the network parameters are continuously updated through the reinforcement learning algorithm. A hybrid reward function is used to enable the agent to continuously learn and optimize the parking trajectory planning strategy. All the above training processes are conducted in a Python environment, and finally, testing and optimization are performed on the highway-env simulation environment.

[0025] In specific implementation, the method of the present invention includes the following steps: The first part concerns the construction of a random parking scenario. Specifically, this invention modifies the configuration of the parking environment in the highway-env library of OpenAI Gym to create a parking lot environment that conforms to real-world conditions, such as... Figure 3 As shown, a scenario with multiple parking spaces is designed, and wall obstacles are set to restrict vehicle movement. Target parking spaces that the autonomous vehicle needs to reach are represented by blocks, and the orientation of the target parking spaces is restricted. Based on this, multiple scenarios of varying complexity are set up, including static obstacles, static vehicles, and dynamic vehicles. This environment encapsulates input and output structures for training and testing reinforcement learning algorithms, and the environment can be modified through parameterized settings. Figure 4 As shown, the encapsulated environment interacts with the DDPG agent. The parking environment receives actions as input and outputs the state and the reward corresponding to the current state.

[0026] Specifically, the environment includes the controlled vehicle (the vehicular vehicle), other static and dynamic vehicles, and other types of obstacles. Vehicle state information is represented by a 6-dimensional vector, such as... Figure 5 As shown, that is These represent position, speed, and orientation angle, respectively. The target parking space status information and the central guide area information are represented by... This indicates that the velocity component is 0. Other obstacles represent their position information. .

[0027] This invention utilizes the OpenAI Gym environment simulation library to create a parking environment. On one hand, any modifications or optimizations only require changes to the internal code of the parking environment, without altering other structures, facilitating debugging and optimization. On the other hand, it can use a general reinforcement learning training framework, enabling rapid deployment and training.

[0028] To enable autonomous vehicles to perform human-like parking maneuvers, this invention implements a mid-area guidance mechanism, such as... Figure 6 As shown, by adaptively providing a guidance area near the target parking space, autonomous vehicles are able to perform parking operations in a human-like manner.

[0029] The second part defines the reinforcement learning problem. In order to connect the constructed parking environment with the reinforcement learning model, this invention defines the task that the method needs to complete, namely, the parking lot initializes the starting position of the self-vehicle, and randomly places static and dynamic vehicles and static obstacles in the parking space and other positions. The self-vehicle's goal is to successfully park in an empty parking space in a human-like manner, without colliding with other objects.

[0030] It includes the following parts: 1) State space: Based on the state information inside the environment, features are preprocessed. That is, with the self-vehicle as the center, other state information is processed into a form relative to the self-vehicle, thereby increasing the generalization of the training scenario.

[0031] 2) Action Space: The reinforcement learning agent outputs actions to the parking environment and then drives the vehicle. To achieve trajectory planning, the action output used in this invention is the vehicle's acceleration and front wheel angle, meaning the action space is a two-dimensional continuous action space. ,in For acceleration, The front wheel steering angle is specifically mapped within the neural network to the range [-1, 1] to suit the network's output. This input motion, based on the vehicle's actual physical characteristics, drives the bicycle model to move within the environment.

[0032]

[0033]

[0034]

[0035]

[0036] in, For the location of your own vehicle; Forward speed; Orientation angle; For acceleration control; The slip angle of the vehicle's center of gravity; This refers to the steering angle of the front wheels.

[0037] The updating of the internal state of the parking environment, i.e., the change of vehicle position, is addressed in this invention using a vehicle kinematics model, which ignores lateral slippage and only allows for control of the front wheels, as parking is a low-speed motion planning problem. This physical model ensures accurate vehicle control and simulation; that is, based on the current vehicle state and control input, the state update for the next moment can be accurately obtained.

[0038] 3) The reward function is set to a mixed type of reward, including sparse reward and per-step reward, and the specific implementation is as follows:

[0039] The sparse rewards include the reward for reaching the finish line, the reward for reaching the intermediate guide area, and the penalty for collisions. The reward for each step is represented as... Specifically defined as:

[0040] The reward for each step is negative, with the goal of minimizing the number of steps required. This indicates the weight of each dimension of data. By setting this weight, the reward function pays more attention to the difference in distance to the destination parking space when the distance is far, and pays more attention to the difference in orientation angle when the distance is close, thereby ensuring that the vehicle parks in the parking space more quickly and accurately.

[0041] The third part describes the construction of the overall network structure of the reinforcement learning algorithm DDPG, as follows: The PPO algorithm is a mainstream single-agent reinforcement learning algorithm suitable for continuous action control. It adopts the classic Actor-Critic network. The Actor-Critic network has the same input state and network structure. The difference is that the Actor uses a policy function to interact with the environment and generate parking input actions, while the Critic uses a value function to evaluate the value of the parking policy and guide the Actor's actions in the next stage.

[0042] Specific implementation as follows Figure 7 As shown: 1) State input: After the parking environment is generated and a guidance area is added, the vehicle state information in it is input into the spatial attention model to realize the attention model of the self vehicle to other vehicles. Then, the information of other obstacles is encoded into a vector of the same dimension using a linear layer network.

[0043] 2) Spatial attention mechanism, which includes three functions: Q, K, and V. This function uses the query Q and key K to calculate an attention score, and applies the normalized score to the value V, where d is the dimension of the query and key.

[0044] In the specific implementation, the Q value is calculated by passing the state of the self-vehicle to the linear layer, and the K and V values ​​are calculated by passing the state of other vehicles to the linear layer. Then, the linear embedding of the self-vehicle's features to other vehicles is obtained through the above formula, thereby realizing the self-vehicle's attention mechanism to other vehicles.

[0045]

[0046]

[0047]

[0048] The feature vector output by the attention model is then processed through two fully connected layers 128 and 64. The output feature vector is then concatenated with the vector obtained after obstacle processing and used as input to the reinforcement learning Actor and Critic networks.

[0049] 3) Actor and Critic Network Layers: The Actor-Critic network has the same input state and network structure, both implemented using two fully connected layers of size 64. After the output of each fully connected layer, it is activated by the Tanh() function, which enables the network to have better nonlinear fitting capabilities. Finally, in the output layer, the Actor network outputs a two-dimensional action, and the Critic network outputs a one-dimensional value.

[0050] Part Four, Reinforcement Learning DDPG utilizes a hybrid type reward function to optimize the training of the network model. Specific details are as follows: In a parking environment, this invention randomizes the initial pose of the autonomous vehicle to increase the robustness of the model and enable it to adapt to different scenarios. The entire training process of the algorithm is shown in Algorithm 1:

[0051] First, initialize the network parameters, experience replay area, and environment. Then, randomly sample actions to collect sufficient experience. Finally, perform training and network updates.

[0052] The details of the hyperparameters related to DDPG during training are as follows:

[0053] Specifically, regarding these parameters, the learning rate of the Actor network is lower than that of the Critic network to ensure stable policy updates. The buffer size refers to the total capacity to store experience data, and the batch size refers to the number of experiences sampled from the buffer each time. This random sampling method ensures sample diversity and guarantees the effectiveness of network parameter updates. DDPG Setting a smaller value is to enable soft updates of the parameters from the Actor and Critic networks to the target Actor and Critic networks.

[0054] This invention does not employ imitation learning for initialization, and abandons the use of expert datasets, thus avoiding unpredictable and extreme cases not covered by expert data. It allows the reinforcement learning agent to autonomously interact with the environment and learn strategies. Compared to methods using only imitation learning or a combination of reinforcement learning and imitation learning, the method proposed in this invention improves model performance and robustness. To enhance the performance of the reinforcement learning agent in complex environments, this invention proposes a hybrid attention mechanism that enables autonomous vehicles to pay attention to the behavior of other moving and static vehicles in the environment. Figure 8 It can be seen that the hybrid attention mechanism effectively improves the training speed and results. Furthermore, to make the planned trajectory more consistent with human driving habits, an intermediate guidance mechanism was implemented, achieving human-like trajectory planning and accelerating the model's training speed. Figure 9 The visualization results demonstrate the role of the guiding region. Finally, comparative and ablation experiments were conducted in the OpenAI gym environment to demonstrate the effectiveness and robustness of the proposed method.

[0055] This invention also provides a human-like parking reinforcement learning trajectory planning system with an attention mechanism, comprising: The parking scenario construction and preprocessing module is used to construct a training environment for simulating parking scenarios. The training information in the parking scenario is represented by vectors. The training information includes the current position, orientation and speed of the autonomous vehicle, the position, orientation and speed of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. The above information is then stored in the form of vectors as the state information of its own environment. The hybrid attention mechanism module is used to construct an attention network based on the training information in the parking scenario, so that the attention network outputs a corresponding feature vector based on the mixed dynamic and static information of the input training information. Specifically, the hybrid attention mechanism module preprocesses the state information, that is, it takes the state information of the autonomous vehicle and other dynamic and static vehicles as input, extracts the information features of dynamic and static obstacles relative to the autonomous vehicle, and then outputs the feature vector as part of the input of the downstream reinforcement learning Actor-Critic network. The intermediate region guidance module is used to train the reinforcement learning model. It incorporates a guidance mechanism based on human preferences into the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. Specifically, the intermediate area guidance module adaptively adds guidance areas to the environment based on the position of the autonomous vehicle and the target parking space in the initialized parking environment. and rewards

[0056] The model training module is used to initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the mixed type reward function, and finally obtain a robust reinforcement learning model. Specifically, the model training module is used to train a reinforcement learning agent model. It interactively collects multiple sets of driving trajectory data based on the OpenAI gym-highway simulation environment. Under the action of a hybrid reward function, the DDPG algorithm optimizes parking trajectory planning to obtain the optimized strategy. ; The hybrid reward function automatically triggers the corresponding reward function based on the selected strategy; specifically, based on the parking task, it will... The state of the environment at any given time is defined as a state. Parking strategy According to the status Receive control commands transition to Moment State and receive a reward This process accumulates until the parking endpoint is reached, at which point the parking strategy must be adjusted. To obtain long-term benefits. .

[0057] The parking trajectory output module is used to output steering wheel angle and acceleration commands based on the training information in the input parking scenario and the trained learning model, so as to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

[0058] Based on the method described in this invention, a vehicle may also be provided, which implements automatic parking based on the method or system described in this invention.

[0059] This invention proposes a human-like parking trajectory planning method and system based on reinforcement learning with an attention mechanism. It addresses some problems of existing reinforcement learning-based methods, such as failing to consider different types of interactions between all agents, ignoring human driving habits, and thus being unsuitable for dynamic, unstructured scenarios. By designing an attention mechanism and implementing an attention network, the system can effectively capture the interactions of heterogeneous traffic information through output feature vectors, improving sample efficiency and model training speed, and guiding autonomous vehicles to plan with a longer-range perspective. The implementation of an adaptive guidance region mechanism enables agents to learn human-like operating strategies, accelerating the training process through intermediate planning points and improving the efficiency and safety of automatic parking. Finally, by continuously training the reinforcement learning model in randomized scenarios, a safer and more efficient automatic parking trajectory planning is achieved, constraining vehicles to park in parking spaces more accurately and human-like.

[0060] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described humanoid parking trajectory planning method based on an attention mechanism.

[0061] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0062] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes the above-described humanoid parking trajectory planning method based on an attention mechanism.

[0063] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0064] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0065] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A human-like parking trajectory planning method based on an attention mechanism, characterized in that, Includes the following steps: S1. Construct a training environment for simulating parking scenarios, and represent the training information in the parking scenarios using vectors. The training information includes the current position, orientation and speed of the autonomous vehicle, the position, orientation and speed of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. S2. Based on the training information in the parking scenario, an attention network is constructed using a hybrid attention mechanism, so that the attention network outputs a corresponding feature vector based on the hybrid dynamic and static information of the input training information; S3. Train the reinforcement learning model and add a guidance mechanism based on human preferences to the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. S4. Initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, and optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the hybrid type reward function. S5. Based on the training information in the input parking scenario, the trained learning model outputs steering wheel angle and acceleration commands to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

2. The human-like parking trajectory planning method based on attention mechanism according to claim 1, characterized in that, The simulated parking scenario constructed is as follows: The state information in the scene includes the current autonomous vehicle, the target parking space, other moving and static vehicles besides the current autonomous vehicle, and static obstacles; each state information is represented using a 6-dimensional vector data. , representing position, speed, and orientation angle information respectively, and distinguishing between static and dynamic vehicles by speed, where the speed parameter in the target parking space vector is 0.

3. The human-like parking trajectory planning method based on attention mechanism according to claim 1, characterized in that, Attention networks are constructed using a hybrid attention mechanism, including: The attention network encodes the current vehicle state and the surrounding static and dynamic vehicle states, extracts information features of dynamic and static obstacles, captures the interaction between heterogeneous traffic information, and constructs the attention network based on the training information in the parking scenario.

4. The human-like parking trajectory planning method based on attention mechanism according to claim 1, characterized in that, The human preference-based guidance mechanism adaptively generates intermediate guidance positions based on human preferences using the initial position of the current vehicle and the target parking space information. An arrival reward mechanism is added to each intermediate guidance position. Based on the current vehicle state, the position weight function is used to determine whether the agent has reached the intermediate guidance position. If so, the arrival reward mechanism is executed to guide the behavior of the model agent.

5. The human-like parking trajectory planning method based on attention mechanism according to claim 2, characterized in that, The DDPG algorithm network structure includes an Actor network and a Critic network. The outputs of the Actor network and the Critic network are different, but the rest of the structure is the same. The inputs to the Actor network and the Critic network are vector information of the autonomous vehicle, the target parking space, other dynamic and static vehicles and static obstacles besides the current autonomous vehicle, and the intermediate guide position. The information of the autonomous vehicle and other dynamic and static vehicles and static obstacles is input into the attention network, which outputs a corresponding feature vector. This feature vector is then concatenated with the other state information. The Actor network and the Critic network are connected and output through two fully connected layers, and are activated by the Tanh() function. The Actor network directly outputs the steering wheel angle and acceleration, and the last layer of the Critic network directly outputs a value.

6. A human-like parking trajectory planning system based on an attention mechanism, characterized in that, include: The parking scenario construction and preprocessing module is used to construct a training environment for simulating parking scenarios. The training information in the parking scenario is represented by vectors. The training information includes the current autonomous vehicle's position, orientation and speed, the positions, orientations and speeds of other vehicles besides the current autonomous vehicle, the target parking space information, and obstacle information. The hybrid attention mechanism module is used to construct an attention network based on the training information in the parking scenario, so that the attention network outputs a corresponding feature vector based on the mixed dynamic and static information of the input training information. The intermediate region guidance module is used to train the reinforcement learning model. It incorporates a guidance mechanism based on human preferences into the model. By guiding the model through intermediate points, the agent of the learning model learns the intermediate parking strategy to obtain a driving trajectory that is more in line with human driving habits. The model training module is used to initialize the Actor network and Critic network of the learning model, train the learning model through the constructed simulated parking scenario, and optimize the agent and intermediate parking strategy of the learning model through the DDPG algorithm under the action of the hybrid type reward function. The parking trajectory output module is used to output steering wheel angle and acceleration commands based on the training information in the input parking scenario and the trained learning model, so as to obtain the automatic parking trajectory planning of the current autonomous vehicle and complete the automatic parking.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the humanoid parking trajectory planning method based on the attention mechanism as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that executes the humanoid parking trajectory planning method based on the attention mechanism according to any one of claims 1 to 5.

Images