Automatic parking method, system and vehicle

Abstract

The embodiment of the application discloses an automatic parking method, system and vehicle. The method comprises the following steps: after obtaining the current state information of the vehicle, obtaining a plurality of action strategies and corresponding total rewards based on the state information and a reinforcement learning network, wherein the total reward is obtained based on the single-step reward corresponding to each action in each action strategy, the single-step reward is obtained based on the estimated vehicle collision information, state information and vibration intensity information after the vehicle executes the corresponding action, and the target action strategy is determined based on the total reward to automatically park based on the target action strategy. Through the above method, based on the state information and the reinforcement learning network, a plurality of action strategies and corresponding total rewards are obtained. Since the single-step reward is obtained based on the vehicle collision information, state information and vibration intensity information, the single-step reward rewards each action based on three dimensions of safety, vehicle state and comfort, so that the vehicle parking action is safer, and the vehicle driving stability in the automatic parking process is improved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to an automatic parking method, system, and vehicle. Background Technology

[0002] With technological advancements and economic development, global car ownership is continuously increasing, especially against the backdrop of accelerated urbanization. More and more families are choosing to purchase private cars to meet their travel needs, leading to a decrease in available space for vehicles and increasing the difficulty of parking. One approach is to use route planning and tracking-based automated parking algorithms: designing parking routes under the constraints of a driving model, and then controlling the vehicle to automatically park based on the parking path. However, this approach still suffers from poor vehicle stability during automated parking. Summary of the Invention

[0003] In view of the above problems, this application proposes an automatic parking method, system, and vehicle to improve the above problems.

[0004] In a first aspect, this application provides an automatic parking method, the method comprising: acquiring current state information of a vehicle; obtaining multiple action strategies and a total reward corresponding to each of the multiple action strategies based on the state information and a reinforcement learning network, wherein the total reward is obtained based on the single-step reward corresponding to each action in each action strategy, and the single-step reward is obtained by estimating the collision information, state information and vibration intensity information of the vehicle after the vehicle performs the corresponding action; and determining a target action strategy based on the total reward, so as to automatically park the vehicle based on the target action strategy.

[0005] Secondly, this application provides an automatic parking system, the system comprising: a data acquisition unit for acquiring current state information of a vehicle, the state information including the vehicle's speed, position, and driving direction; a data calculation unit for obtaining multiple action strategies and a total reward corresponding to each of the multiple action strategies based on the state information and a reinforcement learning network, the total reward being based on the single-step reward corresponding to each action in each action strategy, the single-step reward being obtained by estimating the collision information, state information, and vibration intensity information of the vehicle after the vehicle performs the corresponding action, and determining a target action strategy based on the total reward; and a control unit for automatically parking based on the target action strategy.

[0006] Thirdly, this application provides a vehicle including one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the methods described above.

[0007] Fourthly, this application provides a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run.

[0008] This application provides an automatic parking method, system, vehicle, and storage medium. After acquiring the current state information of the vehicle, multiple action strategies and a total reward corresponding to each of the multiple action strategies are obtained based on the state information and a reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information, and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, a target action strategy is determined for automatic parking based on the target action strategy. The above method enables the generation of multiple action strategies and their corresponding total rewards based on state information and reinforcement learning networks. This allows for the determination of a target action strategy and the control of the vehicle to complete parking based on that strategy. Since the total reward is derived from the single-step reward for each action within each action strategy, and the single-step reward is based on the estimated collision information, state information, and vibration intensity information of the vehicle after executing the corresponding action, the safety level of the action can be determined based on the collision information, the vehicle state can be determined based on the vehicle state information, and the comfort level can be determined based on the vibration intensity information. This allows for the allocation of single-step rewards to each action within each action strategy, with rewards based on three dimensions: safety level, vehicle state, and comfort level. Consequently, the automatic parking process becomes safer and more comfortable, thereby improving the vehicle's driving stability during automatic parking. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of an automatic parking method proposed in this application embodiment;

[0011] Figure 2 This application shows Figure 1 A flowchart of an implementation method proposed in S120;

[0012] Figure 3 This application shows Figure 2 A flowchart of an implementation method proposed in S121;

[0013] Figure 4 A flowchart of an automatic parking method according to another embodiment of this application is shown;

[0014] Figure 5 A schematic diagram of a vibration model of a human sitting posture proposed in an embodiment of this application is shown;

[0015] Figure 6 This paper shows a structural block diagram of an automatic parking system according to an embodiment of the present application;

[0016] Figure 7 A structural block diagram of a vehicle proposed in this application is shown. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] In this application embodiment, the inventors propose an automatic parking method, system, and vehicle. After acquiring the current state information of the vehicle, multiple action strategies and the total reward corresponding to each of the multiple action strategies are obtained based on the state information and a reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information, and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, a target action strategy is determined to automatically park the vehicle based on the target action strategy. The above method enables the generation of multiple action strategies and their corresponding total rewards based on state information and reinforcement learning networks. This allows for the determination of a target action strategy and the control of the vehicle to complete parking based on that strategy. Since the total reward is derived from the single-step reward for each action within each action strategy, and the single-step reward is based on the estimated collision information, state information, and vibration intensity information of the vehicle after performing the corresponding action, the safety level of the action can be determined based on the collision information, the vehicle state can be determined based on the vehicle state information, and the comfort level can be determined based on the vibration intensity information. This allows for the allocation of single-step rewards to each action within each action strategy, with rewards based on three dimensions: safety level, vehicle state, and comfort level. This enables the vehicle to park in a better posture, thereby reducing parking difficulties.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] Reinforcement learning (RL) is a machine learning method typically used to teach an agent how to perform a specific task by interacting with its environment. It can generally be divided into value-function-based methods and policy-based methods. Value-function-based methods learn a value function to evaluate the merits of each action, while policy-based methods learn a policy to select the optimal action.

[0021] Value: In reinforcement learning, value refers to the expected reward that an agent can obtain after adopting a certain strategy.

[0022] Entropy: It can be a variable representing the degree of disorder. Under natural circumstances, things tend to become more disordered, which is a process of increasing entropy. In this application, the higher the entropy value, the more random the strategy and the greater the reward for the target.

[0023] Soft Actor-Critic (SAC): This can be a reinforcement learning algorithm that combines the advantages of policy gradient methods and value function estimation methods, while having good stability and adaptability.

[0024] Strategy: In reinforcement learning, strategy refers to how an agent chooses actions based on its current state and observations when facing its environment, in order to maximize long-term cumulative rewards.

[0025] p-norm: p-norm is a mathematical concept that can be used to measure the length or size of each vector in a vector space. In this application, 2-norm is used. 2-norm is the square root of the sum of the squares of the absolute values ​​of the vector elements.

[0026] Human sitting posture vibration model: This refers to a model that describes the human body's response to vibrations while seated. A typical human sitting posture vibration model includes three input points: linear and angular vibrations at the seat support surface, and linear vibrations at the seat back and foot support surface. Vibration information from these input points is collected by sensors and used to assess the vibration impact on the human body.

[0027] Please see Figure 1 This application provides an automatic parking method, the method comprising:

[0028] S110: Obtain the current status information of the vehicle.

[0029] Among these, vehicle status information can characterize the real-time operating status of the vehicle. The current vehicle status information refers to the collection of data such as the vehicle's speed, position, and direction of travel at the current moment. Vehicle speed refers to the vehicle's real-time moving speed. Position information can include the vehicle's position in two-dimensional space.

[0030] For example, the vehicle's state information can be represented by S, and the vehicle's speed can be represented by v, and the vehicle's speed can be obtained by decomposing it based on the vehicle's coordinate system. x v y The vehicle's position information can be represented by its absolute position in the geodetic coordinate system, using x and y; the vehicle's direction of travel can be represented by its heading angle Φ, which can be obtained by decomposing it into cosΦ and sinΦ based on the geodetic coordinate system; therefore, the vehicle's state information S can be represented as S = {x, y, v}. x v y ,cosΦ,sinΦ}.

[0031] In one approach, when a vehicle is automatically parking, it can acquire current status information of the vehicle, including its speed, location, and direction of travel, through its respective information acquisition devices (such as accelerometers, GPS positioning systems, etc.), and store the acquired status information in a database.

[0032] S120: Based on the state information and the reinforcement learning network, multiple action strategies and the total reward corresponding to each of the multiple action strategies are obtained. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information and vibration intensity information of the vehicle after the vehicle performs the corresponding action.

[0033] Reinforcement learning networks can be used to compute the optimal action strategy for a vehicle during its interaction with its environment. An action strategy can refer to a set of consecutive actions that complete one automatic parking maneuver; each action strategy can include multiple actions. Vehicle collision information can include whether a collision occurred or not. Vehicle vibration intensity information refers to the vibration impact on the human body during automatic parking.

[0034] One approach is to obtain multiple action strategies for a vehicle to complete automatic parking based on state information and reinforcement learning networks. This allows us to obtain multiple actions corresponding to each action strategy, as well as the collision information, state information, and vibration intensity information of the vehicle corresponding to each action. This enables us to obtain the single-step reward for each action, and thus the total reward for each action strategy.

[0035] As a way, such as Figure 2As shown, based on the state information and the reinforcement learning network, multiple action policies and their respective total rewards are obtained, including:

[0036] S121: In each policy generation process, based on the soft action-evaluation algorithm, the state information and the reinforcement learning network, the corresponding actions and multiple single-step rewards for each policy generation process are obtained from the preset action space, and the multiple actions and the multiple single-step rewards correspond one-to-one.

[0037] The strategy generation process refers to the process of determining the action strategy for an automatic parking maneuver based on the current state information. Each strategy generation process may include multiple action search processes. The action search process refers to the process of selecting the corresponding action based on the current state information. The preset action space may include the preset parameter ranges corresponding to vehicle acceleration and front wheel steering angle.

[0038] In one approach, the preset motion space can include multiple preset motion parameters, each of which can include the vehicle's acceleration and front wheel steering angle. The vehicle's front wheel steering angle can be the deflection angle of the vehicle's front wheels with reference to the vehicle's heading angle Φ.

[0039] Optionally, the vehicle's motion parameters can be represented by A, and the vehicle's acceleration can be represented by a. The preset parameter range for the vehicle's acceleration can be [-1.25 m / s²]. 2 1.25m / s 2 The front wheel steering angle can be represented by σ, and the preset parameter range of the vehicle's front wheel steering angle can be [-40°, 40°]; then the vehicle's motion parameter A can be represented as A={a,σ}.

[0040] Optionally, different front wheel steering angles can be sequentially assigned based on an acceleration. For example, as shown in Table 1, action parameter A may include:

[0041] Table 1

[0042] <![CDATA[Acceleration a (m / s 2 )]]> Front wheel steering angle σ (°) Action parameter A 0.2 10 {0.2，10} 0.2 11 {0.2，11} ...... ...... ...... 0.3 10 {0.3，10} 0.3 11 {0.3，11} ...... ...... ......

[0043] In this embodiment, the vehicle's acceleration and front wheel steering angle are set within a certain range, which can prevent significant shaking inside the vehicle due to excessive acceleration or excessive front wheel steering angle during automatic parking. Furthermore, the range of vehicle acceleration and the range of vehicle front wheel steering angle can be adjusted based on vehicle type or multi-vehicle test results.

[0044] One approach is to obtain multiple action policies and the corresponding actions for each action policy based on state information and soft action-evaluation algorithms; another approach is to obtain the single-step reward for each action based on state information, multiple action policies, the corresponding actions for each action policy, and a reinforcement learning network.

[0045] As a way, such as Figure 3 As shown, in each policy generation process, based on the soft action-evaluation algorithm, state information, and reinforcement learning network, the corresponding action and multiple single-step rewards for each policy generation process are obtained from the preset action space, including:

[0046] S1211: During the current action search process, based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network, the actions in the current action search process are obtained from the preset action space.

[0047] One approach is to select action parameters from a preset action space based on the current state information, and obtain the action during the current search process based on the vehicle acceleration and front wheel steering angle corresponding to the selected action parameters.

[0048] Optionally, based on state information, a soft-action evaluation algorithm, and a preset action space, multiple actions can be obtained under the current state information. Then, based on the estimated reference values ​​corresponding to these multiple actions, the action with the largest reference value is selected as the action in the current action search process. The reference value can be obtained based on the single-step reward and entropy value corresponding to the selected action. Therefore, the expression for the soft-action evaluation algorithm can be:

[0049]

[0050] Where, π * R(S) represents updating the action strategy that has already been searched for, where t represents the number of action searches, and R(S) represents the action strategy that has already been searched for. t A t H(π(·|S)) can represent the single-step reward for the action in the t-th action search process, α can represent the entropy regularization coefficient, and H(π(·|S)) can represent the single-step reward for the action in the t-th action search process. t )) can represent the entropy value corresponding to the action in the t-th action search process.

[0051] As one approach, entropy can be calculated using the following expression:

[0052] H(π(·|s′))=-E a logπ(a′|s′)

[0053] Here, s′ can represent the current state information, and a′ can represent the current action parameters.

[0054] In this embodiment, a larger entropy value allows for more exploration of the environment, thus preventing the strategy from converging to a local optimum. Therefore, in the early stages of training the soft-action-evaluation algorithm, α can be set to a larger value to increase the entropy regularization coefficient, thereby allowing for a wider range of action parameters to be selected based on the vehicle's current state information. In the later stages of training, α can be set to a smaller value to reduce or eliminate the entropy regularization coefficient, thereby minimizing the impact of entropy on the selection of corresponding action parameters.

[0055] One approach is to obtain the value corresponding to an action in the current action search process based on state information, the actions in the current action search process, and the reinforcement learning network.

[0056] Optionally, the reinforcement learning network may include a first expression and a second expression. The first expression of the reinforcement learning network can be:

[0057]

[0058] Among them, R(S) t A t Q can represent the single-step reward for the action in the t-th action search process, γ can represent the value discount factor, and Q can represent the single-step reward for the action in the t-th action search process. soft (S t+1 A t+1 ) can represent the value corresponding to the action in the (t+1)th action search process, -αlog(π(A) t+1 |S t+1 )) can represent the entropy value corresponding to the action in the (t+1)th action search process.

[0059] Optionally, the second expression for the reinforcement learning network can be:

[0060] v π (S)=E[R t+1 +γv π (S t+1 )|S t =S]

[0061] Among them, v π (S) can represent the value of the current state, R t+1 γ can represent the single-step reward corresponding to the (t+1)th state information, and v can represent the value discount factor. π (S t+1 ) can represent the value corresponding to the state information at the (t+1)th time.

[0062] In this embodiment, the second expression can be used to calculate the value of all actions under the current state information, while the first expression optimizes the second expression. The first expression can calculate the value of the action in the current action search process under the current state information, and can know the different single-step rewards corresponding to different actions, thereby obtaining the total reward corresponding to multiple action strategies. Therefore, in this embodiment, the first expression is generally selected as the reinforcement learning network to improve the accuracy of the action search process.

[0063] S1212: Based on the actions of the current action search process, obtain the updated state information, collision information and vibration intensity information corresponding to the current action search process, wherein the updated state information is the estimated state information of the vehicle after performing the actions of the action search process.

[0064] One approach is to obtain the updated state information, collision information, and vibration intensity information corresponding to the predicted actions after the current action search process, based on the vehicle's current state information and the actions in the current action search process.

[0065] S1213: Based on the updated state information, collision information, and vibration intensity information corresponding to the current action search process, obtain the single-step reward corresponding to the action in the action search process.

[0066] One approach is to predict the reward for the current action search process based on the updated state information, collision information, and vibration intensity information corresponding to the action in the current action search process, thereby obtaining the single-step reward for the action in the current action search process.

[0067] S1214: Continue to the next action search process. In the next action search process, the action of the next action search process is obtained by the single-step reward, update state information, soft action-evaluation algorithm and reinforcement learning network corresponding to the current action search process, until the update state information is the same as the target state information, which is the state information when the vehicle automatically parks to the target position.

[0068] In one approach, during each policy generation process, the single-step reward and updated state information corresponding to the current action search process can be input into the soft action-evaluation algorithm and reinforcement learning network. After each action in the action search process is generated, the single-step reward and updated state information corresponding to the action in that action search process will be obtained until the updated state information and the target state information are the same.

[0069] In this embodiment of the application, before the updated state information and target state information are the same, the corresponding actions and multiple single-step rewards for each policy generation process are obtained from the preset action space based on the soft action-evaluation algorithm, the state information and the reinforcement learning network, thereby obtaining the action policy.

[0070] S122: Based on the multiple actions corresponding to each strategy generation process, generate action strategies corresponding to each strategy generation process to obtain the multiple action strategies.

[0071] One approach is to add timestamps to multiple actions corresponding to each policy generation process based on the order in which they are generated, and then sort them based on the timestamps. This allows for the generation of consecutive actions based on multiple actions and their corresponding timestamps, thereby obtaining the corresponding action policy.

[0072] As another approach, multiple actions corresponding to each policy generation process can be prioritized based on their generation order and sorted according to their priorities to obtain consecutive actions based on multiple actions and their corresponding priorities, thereby obtaining the corresponding action policy.

[0073] S123: Based on the multiple single-step rewards corresponding to each policy generation process, obtain the total reward of the action policy corresponding to each policy generation process, so as to obtain the total reward corresponding to each of the multiple action policies.

[0074] One approach is to obtain the total reward of the action policy corresponding to each policy generation process based on the sum of the rewards of multiple single steps corresponding to each policy generation process.

[0075] As another approach, the single-step rewards corresponding to multiple strategy generation processes can be deweighted. The weight of the single-step reward corresponding to the action generated in the early stage of parking is relatively small, the weight of the single-step reward corresponding to the action generated in the middle stage of parking is medium, and the weight of the single-step reward corresponding to the action generated in the later stage of parking is relatively large. Then, based on the sum of multiple single-step rewards, the total reward of the action strategy corresponding to each strategy generation process can be obtained, thus obtaining the total reward corresponding to each of the multiple action strategies.

[0076] S130: Based on the total reward, determine a target action strategy to automatically park the vehicle based on the target action strategy.

[0077] The target action policy can be the optimal action policy among multiple action policies.

[0078] One approach is to select the action strategy with the largest total reward from among multiple action strategies as the target action strategy.

[0079] Optionally, based on multiple actions corresponding to the target action strategy, there are multiple action parameters. Based on the multiple action parameters, the vehicle's acceleration and front wheel steering angle corresponding to each action can be determined, thereby controlling the vehicle to perform automatic parking based on the vehicle's acceleration and front wheel steering angle corresponding to each action.

[0080] This embodiment provides an automatic parking method. After acquiring the current state information of the vehicle, multiple action strategies and their corresponding total rewards are obtained based on the state information and a reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information, and vibration intensity information of the vehicle after it performs the corresponding action. Based on the total reward, a target action strategy is determined for automatic parking. The above method enables the generation of multiple action strategies and their corresponding total rewards based on state information and reinforcement learning networks. This allows for the determination of a target action strategy and the control of the vehicle to complete parking based on that strategy. Since the total reward is derived from the single-step reward for each action within each action strategy, and the single-step reward is based on the estimated collision information, state information, and vibration intensity information of the vehicle after executing the corresponding action, the safety level of the action can be determined based on the collision information, the vehicle state can be determined based on the vehicle state information, and the comfort level can be determined based on the vibration intensity information. This allows for the allocation of single-step rewards to each action within each action strategy, with rewards based on three dimensions: safety level, vehicle state, and comfort level. Consequently, the automatic parking process becomes safer and more comfortable, thereby improving the vehicle's driving stability during automatic parking.

[0081] Please see Figure 4 This application provides an automatic parking method, the method comprising:

[0082] S210: During the current action search process, based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network, the actions in the current action search process are obtained from the preset action space.

[0083] S220: Based on the actions of the current action search process, obtain the updated state information, collision information and vibration intensity information corresponding to the current action search process, wherein the updated state information is the estimated state information of the vehicle after performing the actions of the action search process.

[0084] S230: Based on the updated state information and the target state information corresponding to the current action search process, obtain the first reference reward corresponding to the action, wherein the greater the difference between the updated state information and the target state information, the smaller the first reference reward.

[0085] The updated status information can include the vehicle's updated speed, updated location, and updated direction of travel. The target status information can include the vehicle's target speed, target location, and target direction of travel. The first reference reward can provide feedback on the difference between the vehicle's updated status information and the target status information.

[0086] As one approach, state gap information corresponding to the current action search process can be obtained based on the updated state information and target state information. First state gap information can be obtained based on the updated speed and target speed; second state gap information can be obtained based on the updated position and target position; third state gap information can be obtained based on the updated driving direction and target driving direction; and first reference reward corresponding to the action can be obtained based on the first state gap information, second state gap information, and third state gap information.

[0087] In this embodiment, a method for calculating the norm between updated state information and target state information can be used as the first reference reward. That is, the first state gap information can represent the norm between the updated speed and the target speed, the second state gap information can represent the norm between the updated position and the target position, and the third state gap information can represent the norm between the updated driving direction and the target driving direction. Therefore, the first reference reward is obtained based on the negative of the sum of the first, second, and third state gap information. The formula for the first reference reward can then be expressed as:

[0088]

[0089] Among them, R pos S can represent the first reference reward, and S can represent the updated state information, S = [x, y, v]. x v y [cosΦ, sinΦ]; S g It can represent target state information, S g =[x g y g v xg v yg cosΦ g sinΦ g ].

[0090] One approach is to use the p-norm, where p refers to the sum of the p-th powers of the absolute values ​​of the vector elements, raised to the power of 1 / p. W i = [1, 1, 0, 0, 0.02, 0.2], p = 2; that is, the 2-norm can be used to calculate the first reference reward.

[0091] S240: Based on the collision information and preset rules corresponding to the current action search process, a second reference reward is obtained for the action, and the second reference reward represents the safety level of the action.

[0092] Among them, the preset rules can provide a second reference reward based on whether a vehicle collision occurs or not.

[0093] As one approach, if the collision information indicates that the vehicle has been in a collision, the second reference reward is determined to be the first preset value; if the collision information indicates that the vehicle has not been in a collision, the second reference reward is determined to be the second preset value, and the first preset value is less than the second preset value.

[0094] Optionally, the first and second preset values ​​can be adjusted based on simulation training and experience.

[0095] For example, the first preset value can be set to -5, and the second preset value can be set to 0; that is, if the collision information indicates that the vehicle has been in a collision, the second reference reward is determined to be -5; if the collision information indicates that the vehicle has not been in a collision, the second reference reward is determined to be 0. The formula for the second reference reward can then be expressed as:

[0096]

[0097] In this embodiment of the application, the safety of parking needs to be considered during the automatic parking process. Therefore, a second reference reward for the action can be obtained based on the vehicle's collision information, thereby increasing the reward dimension.

[0098] S250: Based on the vibration intensity information corresponding to the current action search process, a third reference reward corresponding to the action is obtained, wherein the third reference reward characterizes the comfort level of the action.

[0099] Among them, the vibration intensity information of the vehicle can be the vibration impact on the human body in the x and y directions.

[0100] As a way, such as Figure 5 As shown, the vibration intensity of the human body in a sitting position can be obtained from the x and y directions based on the human body sitting posture vibration model, and used as vibration intensity information.

[0101] One approach is to obtain reference vibration intensity information, which is the vibration intensity information corresponding to the previous action search process. Based on the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, a third reference reward is obtained for that action. The greater the difference between the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, the smaller the third reference reward.

[0102] Optionally, the negative of the sum of the differences and influence values ​​between the reference intensity information in the x-direction and the vibration intensity information corresponding to the current action search process, and the differences and influence values ​​between the reference intensity information in the y-direction and the vibration intensity information corresponding to the current action search process, can be obtained as the third reference reward. The formula for the third reference reward can then be expressed as:

[0103]

[0104] Where, k δ and k β It can be used to represent the weights of vibration intensity in the x and y directions, and k δ and k β It can be set to 0.05; It can be used to represent vibration intensity information in the x-direction. It can be used to represent vibration intensity information in the y-direction.

[0105] In this embodiment of the application, the passenger comfort needs to be considered during the automatic parking process. Therefore, a third reference reward for the action can be obtained based on the vehicle's vibration intensity information, thereby adding another reward dimension.

[0106] S260: Based on the first reference reward, second reference reward and third reference reward corresponding to the action, obtain the single-step reward corresponding to the action.

[0107] In one approach, a first weight can be assigned to the first reference reward, a second weight to the second reference reward, and a third weight to the third reference reward. Based on the first reference reward, the first weight of the first reference reward, the second weight of the second reference reward, the third weight of the third reference reward, and the third weight of the third reference reward, the single-step reward corresponding to the action can be obtained.

[0108] As one approach, the formula for single-step reward can be expressed as:

[0109] R=λ1R pos +λ2R safe +λ3R com

[0110] Wherein, λ1 represents the first weight, λ2 represents the second weight, and λ3 represents the third weight. The sum of the first weight, the second weight, and the third weight is 1, and the first weight, the second weight, and the third weight can be adjusted based on training.

[0111] This embodiment provides an automatic parking method that, through the aforementioned approach, allows for the generation of multiple action strategies and their corresponding total rewards based on state information and a reinforcement learning network. This enables the determination of a target action strategy and the control of the vehicle to complete parking based on that strategy. Since the total reward is derived from the single-step reward for each action within each action strategy, and the single-step reward is based on the estimated collision information, state information, and vibration intensity information of the vehicle after executing the corresponding action, the safety level of the action can be determined based on the collision information, the vehicle state can be determined based on the vehicle state information, and the comfort level can be determined based on the vibration intensity information. This allows for the allocation of single-step rewards to each action within each action strategy, with the single-step rewards based on three dimensions: safety level, vehicle state, and comfort level. This makes the automatic parking process safer and more comfortable, thereby improving the vehicle's driving stability during automatic parking. Furthermore, in this embodiment, a first reference reward can be obtained based on the updated state information and target state information corresponding to the action in the current search process; a second reference reward can be obtained based on the collision information corresponding to the current action search process; and a third reference reward can be obtained based on the reference vibration information and the current vibration information corresponding to the action in the current search process, thereby obtaining a single-step reward, which increases the dimension of the reward and improves the efficiency of reinforcement learning network training.

[0112] Please see Figure 6 This application provides an automatic parking system 800, the system 800 comprising:

[0113] The data acquisition unit 810 is used to acquire the current status information of the vehicle, including the vehicle's speed, position, and direction of travel.

[0114] The data calculation unit 820 is used to obtain multiple action strategies and the total reward corresponding to each of the multiple action strategies based on the state information and the reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, the target action strategy is determined.

[0115] Control unit 830 is used for automatic parking based on the target action strategy.

[0116] In one approach, the data computation unit 820 is used to, in each policy generation process, obtain, based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network, the actions corresponding to each policy generation process and multiple single-step rewards from a preset action space, wherein the multiple actions and the multiple single-step rewards correspond one-to-one; generate action policies corresponding to each policy generation process based on the multiple actions corresponding to each policy generation process, to obtain the multiple action policies; and obtain the total reward of the action policies corresponding to each policy generation process based on the multiple single-step rewards corresponding to each policy generation process, to obtain the total reward corresponding to each of the multiple action policies.

[0117] In one approach, the data computing unit 820 is used to, during the current action search process, obtain the action in the current action search process from the preset action space based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network; based on the action in the current action search process, obtain the updated state information, collision information, and vibration intensity information corresponding to the current action search process, wherein the updated state information is the estimated state information of the vehicle after performing the action in the action search process; based on the updated state information, collision information, and vibration intensity information corresponding to the current action search process, obtain the single-step reward corresponding to the action in the action search process; continue to the next action search process, and in the next action search process, obtain the action in the next action search process based on the single-step reward, updated state information, the soft action-evaluation algorithm, and the reinforcement learning network corresponding to the current action search process, until the updated state information is the same as the target state information, wherein the target state information is the state information of the vehicle when it automatically parks to the target position.

[0118] In one approach, the data calculation unit 820 is used to obtain a first reference reward for the action based on the updated state information and the target state information corresponding to the current action search process, wherein the greater the difference between the updated state information and the target state information, the smaller the first reference reward; to obtain a second reference reward for the action based on the collision information and preset rules corresponding to the current action search process, wherein the second reference reward represents the safety level of the action; to obtain a third reference reward for the action based on the vibration intensity information corresponding to the current action search process, wherein the third reference reward represents the comfort level of the action; and to obtain a single-step reward for the action based on the first, second, and third reference rewards corresponding to the action.

[0119] In one manner, the data calculation unit 820 is used to determine the second reference reward as a first preset value if the collision information indicates that the vehicle has been in a collision; and to determine the second reference reward as a second preset value if the collision information indicates that the vehicle has not been in a collision, wherein the first preset value is less than the second preset value.

[0120] In one approach, the data calculation unit 820 is used to obtain reference vibration intensity information, which is the vibration intensity information corresponding to the previous action search process; based on the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, a third reference reward corresponding to the action is obtained, wherein the greater the difference between the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, the smaller the third reference reward.

[0121] Optionally, the preset action space includes the preset parameter ranges corresponding to the vehicle acceleration and the front wheel steering angle, respectively.

[0122] The following will combine Figure 7 This application describes one type of vehicle.

[0123] Please see Figure 7 Based on the above-described automatic parking method and system, this application embodiment also provides another vehicle 100 capable of executing the aforementioned automatic parking method. The vehicle 100 includes a processor 102, a memory 104, and a communication module 106. The memory 104 stores a program capable of executing the contents of the aforementioned embodiments, and the processor 102 can execute the program stored in the memory 104.

[0124] The processor 102 may include one or more processing cores. The processor 102 connects to various parts within the vehicle 100 using various interfaces and lines, and performs various functions and processes data of the vehicle 100 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one of the following hardware forms: a Neural Network Processing Unit (NPU), a Digital Signal Processing Unit (DSP), a Field-Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA). The processor 102 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Neural Network Processing Unit (NPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; the NPU handles multimedia data such as video and images; and the modem handles wireless communication. It is understandable that the aforementioned modem may not be integrated into the processor 102, but may be implemented using a separate communication chip.

[0125] The memory 104 may include random access memory (RAM), read-only memory (ROM), and double data rate synchronous dynamic random access memory (DDR). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the vehicle 100 (such as phonebooks, audio and video data, chat log data, etc.).

[0126] This application provides a computer-readable storage medium 1000. The computer-readable storage medium 1000 stores program code, which can be called by a processor to execute the methods described in the above method embodiments.

[0127] The computer-readable storage medium 1000 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1000 has storage space for program code 1010 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 1010 may be compressed, for example, in a suitable form.

[0128] In summary, the automatic parking method, system, vehicle, and storage medium provided in this application, after acquiring the current state information of the vehicle, obtain multiple action strategies and a total reward corresponding to each of the multiple action strategies based on the state information and a reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information, and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, a target action strategy is determined for automatic parking based on the target action strategy. The above method enables the generation of multiple action strategies and their corresponding total rewards based on state information and reinforcement learning networks. This allows for the determination of a target action strategy and the control of the vehicle to complete parking based on that strategy. Since the total reward is derived from the single-step reward for each action within each action strategy, and the single-step reward is based on the estimated collision information, state information, and vibration intensity information of the vehicle after executing the corresponding action, the safety level of the action can be determined based on the collision information, the vehicle state can be determined based on the vehicle state information, and the comfort level can be determined based on the vibration intensity information. This allows for the allocation of single-step rewards to each action within each action strategy, with rewards based on three dimensions: safety level, vehicle state, and comfort level. Consequently, the automatic parking process becomes safer and more comfortable, thereby improving the vehicle's driving stability during automatic parking.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automatic parking method, characterized in that, The method includes: Obtain the current status information of the vehicle; Based on the state information and the reinforcement learning network, multiple action strategies and the total reward corresponding to each of the multiple action strategies are obtained. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, a target action strategy is determined for automatic parking.

2. The method according to claim 1, characterized in that, Each action policy includes multiple actions. Based on the state information and the reinforcement learning network, the multiple action policies and the total reward corresponding to each of the multiple action policies are obtained, including: In each policy generation process, based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network, the corresponding actions and multiple single-step rewards for each policy generation process are obtained from the preset action space, and the multiple actions and the multiple single-step rewards correspond one-to-one. Based on the multiple actions corresponding to each strategy generation process, an action strategy corresponding to each strategy generation process is generated to obtain the multiple action strategies. Based on the multiple single-step rewards corresponding to each strategy generation process, the total reward of the action strategy corresponding to each strategy generation process is obtained, so as to obtain the total reward corresponding to each of the multiple action strategies.

3. The method according to claim 2, characterized in that, Each policy generation process includes multiple action search processes. In each policy generation process, based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network, the corresponding action and multiple single-step rewards are obtained from the preset action space, including: During the current action search process, the actions in the current action search process are obtained from the preset action space based on the soft action-evaluation algorithm, the state information, and the reinforcement learning network. Based on the actions of the current action search process, the updated state information, collision information and vibration intensity information corresponding to the current action search process are obtained. The updated state information is the estimated state information of the vehicle after performing the actions of the action search process. Based on the updated state information, collision information, and vibration intensity information corresponding to the current action search process, the single-step reward corresponding to the action in the action search process is obtained; The process continues into the next action search process. In the next action search process, the action for the next action search process is obtained by using the single-step reward, updated state information, the soft action-evaluation algorithm, and the reinforcement learning network corresponding to the current action search process, until the updated state information is the same as the target state information, which is the state information when the vehicle automatically parks to the target position.

4. The method according to claim 3, characterized in that, The step of obtaining the single-step reward corresponding to the action in the action search process based on the updated state information, collision information, and vibration intensity information corresponding to the current action search process includes: Based on the updated state information and the target state information corresponding to the current action search process, a first reference reward corresponding to the action is obtained, wherein the greater the difference between the updated state information and the target state information, the smaller the first reference reward; Based on the collision information and preset rules corresponding to the current action search process, a second reference reward is obtained for the action, and the second reference reward represents the safety level of the action. Based on the vibration intensity information corresponding to the current action search process, a third reference reward is obtained for the action, and the third reference reward represents the comfort level of the action; Based on the first reference reward, second reference reward, and third reference reward corresponding to the action, the single-step reward corresponding to the action is obtained.

5. The method according to claim 4, characterized in that, Based on the collision information and preset rules corresponding to the current action search process, a second reference reward corresponding to the action is obtained, including: If the collision information indicates that the vehicle has been involved in a collision, the second reference reward is determined to be the first preset value; If the collision information indicates that the vehicle has not been involved in a collision, the second reference reward is determined to be the second preset value, where the first preset value is less than the second preset value.

6. The method according to claim 4, characterized in that, Based on the vibration intensity information corresponding to the current action search process, a third reference reward corresponding to the action is obtained, including: Obtain reference vibration intensity information, which is the vibration intensity information corresponding to the previous action search process; Based on the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, a third reference reward is obtained for the action. The greater the difference between the reference vibration intensity information and the vibration intensity information corresponding to the current action search process, the smaller the third reference reward.

7. The method according to claim 2, characterized in that, The preset motion space includes the preset parameter ranges corresponding to the vehicle's acceleration and front wheel steering angle, respectively.

8. An automatic parking system, characterized in that, The system includes: The data acquisition unit is used to acquire the current status information of the vehicle, including the vehicle's speed, position, and direction of travel. The data calculation unit is used to obtain multiple action strategies and the total reward corresponding to each of the multiple action strategies based on the state information and the reinforcement learning network. The total reward is obtained based on the single-step reward corresponding to each action in each action strategy. The single-step reward is obtained by estimating the collision information, state information and vibration intensity information of the vehicle after the vehicle performs the corresponding action. Based on the total reward, the target action strategy is determined. A control unit for automatically parking based on the target action strategy.

9. A vehicle, characterized in that, Includes one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the method described in any one of claims 1-7 is executed when the program code is run.

Images