Intelligent driving decision-making system and method based on reinforcement learning

Abstract

The invention provides an intelligent driving decision-making system and method based on reinforcement learning. The method comprises the following steps: mapping a decision-making action into an action space of automatic parking based on a parking control instruction; all reward factors of parking operation in the action space are determined, and then the reward track of the automobile after automatic parking is determined according to the instant reward of each reward factor; the state space of the sensor noise when the automobile is automatically parked is determined, then track matching is conducted on the parking control instruction according to the state space and the action space of the sensor noise, and the anti-interference performance of intelligent driving is obtained; and according to the reward track and anti-interference performance, determining a deviation track of the automobile after automatic parking, and when the deviation track is lower than a preset parking deviation threshold value, performing adaptive constraint adjustment on a decision action of the automobile in automatic parking. On the basis of the scheme, constraint adjustment of the automatic parking action of the automobile can be achieved, and therefore the environmental adaptability of automatic parking of the automobile can be improved.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning driving technology, and more specifically, to an intelligent driving decision-making system and method based on reinforcement learning. Background Art

[0002] In the development of intelligent driving technology, the optimization of decision-making actions is the key to improving the safety and efficiency of autonomous driving systems. Traditional intelligent driving decision-making methods usually rely on predefined rules and models, which are often unable to effectively cope with changing situations in complex and dynamic driving environments. In order to overcome this limitation, reinforcement learning has gradually become an important technology for solving intelligent driving decision-making problems. Combining intelligent decision-making actions with reinforcement learning not only improves the flexibility and accuracy of the decision-making process, but also can effectively cope with complex driving scenarios.

[0003] In traditional automatic parking systems, they often rely on fixed decision-making models and preset operating rules, and it is difficult to flexibly respond to complex and dynamic parking environments. Fixed decision-making models and preset operating rules may perform well under specific conditions, but when encountering uncertainty or emergencies, the car may not be able to respond optimally, resulting in parking failure or even safety hazards. By dynamically adjusting the constraints of automatic parking actions, the car can make adaptive decisions based on real-time environmental feedback (such as obstacle location, road conditions, and surrounding traffic flow). This adjustment mechanism enables the system to better handle different parking scenarios, thereby improving environmental adaptability. Therefore, how to achieve constraint adjustment of the car's automatic parking action, thereby improving the environmental adaptability of the car's automatic parking, is a difficult problem faced by Industrial Street. Summary of the invention

[0004] The present application provides an intelligent driving decision-making system and method based on reinforcement learning, which can realize the constraint adjustment of the automatic parking action of the automobile, thereby improving the environmental adaptability of the automatic parking of the automobile.

[0005] In a first aspect, the present application provides an automatic parking control method based on reinforcement learning, comprising: When the car issues a parking control command to perform automatic parking in intelligent driving, real-time monitoring of the car's automatic parking decision action is performed, and mapping the decision action into an automatic parking action space based on the parking control command; A reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, and then determines the reward trajectory of the car after automatic parking based on the immediate rewards of each reward factor; Determining the state space of the sensor noise of the vehicle during automatic parking by using the sensor noise information of the vehicle during automatic parking, and then performing trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space, so as to obtain the anti-interference performance of the parking operation of the vehicle during automatic parking; The deviation trajectory of the vehicle after automatic parking is determined according to the reward trajectory and the anti-interference performance, and the decision-making action of the vehicle in automatic parking is adaptively constrained and adjusted based on the deviation trajectory.

[0006] In some embodiments, mapping the decision action to an action space of automatic parking based on the parking control instruction specifically includes: Obtain parking control instructions for the car during automatic parking; Decomposing the decision action based on the parking control instruction to obtain all parking actions of automatic parking; The action space of automatic parking is constructed based on all parking actions.

[0007] In some embodiments, the reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, specifically including: Get the driving task of the car during automatic parking; A reinforcement learning-based reward shaping mechanism obtains all task influencing factors of the parking operation from the driving task; Extracting all parking actions of the parking operation from the action space, and then determining the task distance between each parking action; Based on all task distances, reward factors are screened out from all task influencing factors, thereby obtaining all reward factors for the parking operation in the action space.

[0008] In some embodiments, determining the reward trajectory of the vehicle after automatic parking based on the instant rewards of various reward factors specifically includes: Determine the instant reward for each reward factor in the parking maneuver; Filter out the concentrated value of rewards in parking operation from all instant rewards; Determine the reward volatility for parking maneuvers based on all instant rewards; A reward prediction is performed on the initial parking trajectory of the car according to the reward concentration value and the reward volatility to obtain a reward trajectory of the car after automatic parking.

[0009] In some embodiments, determining the state space of the sensor noise of the vehicle during automatic parking by using the sensor noise information of the vehicle during automatic parking specifically includes: Obtain sensor noise information and the vehicle's own status during automatic parking; The sensor noise information and the self-state are combined based on state representation learning in reinforcement learning to generate the representation quality of each parking action of the automobile during automatic parking; The state space of the sensor noise of the car during automatic parking is constructed based on all the characterization qualities.

[0010] In some embodiments, performing trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space to obtain the anti-interference performance of the parking operation of the automobile during automatic parking specifically includes: For each parking action during automatic parking, determining a control deviation of the parking operation according to a characterization quality corresponding to the parking action in the state space of the sensor noise; determining a decision degree of a parking action according to the driving decision in the action space; Based on the control deviation and the decision degree, spatial matching is performed on the driving trajectory of the vehicle after executing the parking control command to obtain the noise matching degree between the parking action and the sensor noise information, and then the noise matching degree between each parking action and the sensor noise information during automatic parking is obtained; The overall noise adaptation determines the vehicle's resistance to interference during automatic parking maneuvers.

[0011] In some embodiments, determining the deviation trajectory of the vehicle after automatic parking according to the reward trajectory and the anti-interference performance specifically includes: Determining a negative rate of a decision action according to the anti-interference performance; Determine a decision loss of a decision action by using the reward trajectory and the interference resistance; The reward trajectory is predicted to have a deviation according to the negative rate and the decision loss, so as to obtain a deviation trajectory of the car after automatic parking.

[0012] In a second aspect, the present application provides an intelligent driving decision system based on reinforcement learning, including a parking control unit, wherein the parking control unit includes: A monitoring module, for monitoring the decision action of the automatic parking of the automobile in real time when the automobile issues a parking control command for automatic parking in intelligent driving, and mapping the decision action into an action space of automatic parking based on the parking control command; A processing module, configured to extract all reward factors of the parking operation from the action space based on a reward shaping mechanism of reinforcement learning, and then determine a reward trajectory of the vehicle after automatic parking based on the instantaneous rewards of each reward factor; The processing module is also used to determine the state space of the sensor noise of the automobile during automatic parking through the sensor noise information of the automobile during automatic parking, and then perform trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space, so as to obtain the anti-interference performance of the parking operation of the automobile during automatic parking; An execution module is used to determine a deviation trajectory of the vehicle after automatic parking according to the reward trajectory and the anti-interference performance, and to perform adaptive constraint adjustment on the decision-making action of the vehicle in automatic parking based on the deviation trajectory.

[0013] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned reinforcement learning-based automatic parking control method.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned reinforcement learning-based automatic parking control method when executed.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: In a reinforcement learning-based intelligent driving decision system and method provided by the present application, when a car issues a parking control command to perform automatic parking in intelligent driving, the decision action of the car's automatic parking is monitored in real time, and the decision action is mapped to the action space of automatic parking based on the parking control command; a reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, and then the reward trajectory of the car after automatic parking is determined by the immediate reward of each reward factor; the state space of the sensor noise of the car during automatic parking is determined by the sensor noise information of the car during automatic parking, and then the parking control command is track-matched according to each characterization quality in the state space of the sensor noise and the driving decision in the action space to obtain the anti-interference of the parking operation of the car during automatic parking; the deviation trajectory of the car after automatic parking is determined based on the reward trajectory and the anti-interference, and the decision action of the car in automatic parking is adaptively constrained and adjusted based on the deviation trajectory.

[0016] It can be seen that in this application, the deviation trajectory of the car after automatic parking is determined according to the reward trajectory and the anti-interference performance, and the decision-making action of the car in automatic parking is adaptively constrained and adjusted based on the deviation trajectory; firstly, in the automatic parking system, the reward trajectory can be determined to obtain the parking trajectory after the decision-making action and environmental feedback during automatic parking, thereby providing a benign feedback mechanism for the decision-making process of automatic parking. By real-time monitoring of the instant reward of each decision action, the car can evaluate its effectiveness and safety in a specific environment. The reward trajectory not only provides a quantitative standard for action selection, but also ensures that the car will not be biased when optimizing decisions. Keep away from the safety range, ensuring that when executing automatic parking, the actions that can reduce adverse events (such as collisions, scratches, etc.) are given priority; then, the anti-interference performance is determined to obtain the ability of the parking operation to maintain normal operation, stability and accuracy under different sensor noise information conditions. By accurately matching the state space and decision-making actions of the sensor noise, the car can adjust the parking strategy in real time according to the actual environmental conditions, so that the vehicle can safely and effectively perform parking operations in various complex scenarios; in summary, based on the above scheme, the constraint adjustment of the automatic parking action of the car can be realized, thereby improving the environmental adaptability of the automatic parking of the car. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is an exemplary flow chart of an automatic parking control method based on reinforcement learning according to some embodiments of the present application; Figure 2 It is a parking control logic diagram shown in some embodiments of the present application; Figure 3 is a schematic diagram of a flow chart of determining a state space of sensor noise according to some embodiments of the present application; Figure 4 is a schematic diagram of the structure of a parking control unit according to some embodiments of the present application; Figure 5 It is a structural diagram of a computer device for implementing an automatic parking control method based on reinforcement learning according to some embodiments of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to better understand the technical solution of the present application, the technical solution of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] refer to Figure 1 , which is an exemplary flow chart of an automatic parking control method based on reinforcement learning according to some embodiments of the present application. The automatic parking control method based on reinforcement learning mainly includes the following steps: In step 101, when a car issues a parking control command for automatic parking in intelligent driving, the decision action of the car for automatic parking is monitored in real time, and the decision action is mapped into an action space for automatic parking based on the parking control command.

[0022] It should be noted that in the present application, the parking control instruction is a command generated by the car based on environmental perception and task requirements. The corresponding control instruction can be generated from the automatic parking system of the car. The parking control instruction includes parking position, vehicle speed, direction control and braking control. The parking position refers to the geometry and position of the predetermined parking area; vehicle speed refers to adjusting the vehicle's driving speed according to the complexity of the parking task and environmental conditions; direction control refers to the steering angle and steering mode (such as forward parking, reversing); braking control refers to controlling the vehicle's braking action during the parking process to ensure smooth parking.

[0023] In specific implementation, when the car issues a parking control command for automatic parking during intelligent driving, the car's central console can be used to monitor the car's automatic parking decision-making actions in real time; among them, the decision-making action refers to the initial parking control command generated in the car's automatic driving central console, and the instructions in the decision-making action directly act on the vehicle's control system to determine the vehicle's next movement behavior, such as steering, acceleration, deceleration, braking, etc., to complete specific driving tasks.

[0024] In some embodiments, reference Figure 2 , which is a parking control logic diagram according to some embodiments of the present application, which includes multiple key components in the parking control logic and the connection relationship between them. Figure 2The body control module (BCM) is located at the center of the vehicle. As the control center of the entire parking control logic, BCM is responsible for receiving and processing signals from various sensors and switches. BCM7 is connected to the ignition switch 1, slope sensor 2, brake pedal 3, speed sensor 4, TCU 5, ESC controller 6, front brake 8, rear brake 9, parking brake 10, brake tail light 11, instrument 12, CAN bus 13 and other components through different lines. These connections indicate that BCM can monitor and control key functions of the vehicle, such as braking, lighting and vehicle stability.

[0025] exist Figure 2 On the left side, the ignition switch (TCU) is connected to the BCM through line 5, which means that the BCM can detect the ignition status of the vehicle. At the same time, the front brake and the rear brake are connected to the BCM through lines 8 and 9 respectively, which means that the BCM can control the working status of the brake to achieve the parking function. In addition, the parking brake is connected to the BCM through line 10, which further emphasizes the core role of the BCM in parking control. The brake taillight is connected to the BCM through line 11 to ensure that the taillight can be lit in time when braking, thereby improving driving safety.

[0026] exist Figure 2 On the right side, the slope sensor is connected to the BCM via line 2, which enables the parking control logic to sense the slope environment of the vehicle and provide appropriate braking support when parking. The ESC controller is connected to the BCM via line 6, which shows that the BCM can communicate with the vehicle's electronic stability control logic to ensure the stability of the vehicle under various driving conditions. Finally, the CAN bus is connected to the BCM via line 12, which shows that the entire logic communicates through the network, which not only improves the efficiency of data transmission, but also enhances the flexibility and scalability of the parking control logic. Overall, the figure shows a complex and sophisticated parking control logic, which ensures the parking safety of the vehicle in various situations through the coordinated work of multiple components.

[0027] In some embodiments, mapping the decision action to the action space of automatic parking based on the parking control instruction can be achieved by the following steps: Obtain parking control instructions when the car is automatically parked; Decomposing the decision action based on the parking control instruction to obtain all parking actions of automatic parking; The action space of automatic parking is constructed based on all parking actions.

[0028] It should be noted that, in the present application, the action space is the set of all parking actions of the car during automatic parking, which supports the system to select appropriate actions to execute under different environmental conditions; in specific implementation, first, the parking control instructions of the car during automatic parking are obtained; then, the action feature extraction algorithm can be used to extract the control actions corresponding to each decision step of the parking control instructions from the decision actions as the parking actions, and all parking actions of automatic parking can be obtained; finally, all parking actions (steering angle, acceleration, braking force) can be arranged in chronological order as the action space of automatic parking.

[0029] In step 102, a reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, and then determines the reward trajectory of the car after automatic parking based on the immediate rewards of each reward factor.

[0030] In some embodiments, the reward shaping mechanism based on reinforcement learning can be implemented by extracting all reward factors of the parking operation from the action space using the following steps: Get the driving task of the car during automatic parking; A reinforcement learning-based reward shaping mechanism obtains all task influencing factors of the parking operation from the driving task; Extracting all parking actions of the parking operation from the action space, and then determining the task distance between each parking action; Based on all task distances, reward factors are screened out from all task influencing factors, thereby obtaining all reward factors for the parking operation in the action space.

[0031] It should be noted that in this application, the reward factor refers to the positive feedback signal that affects the decision of the car console when performing a specific driving task, and the reward factor is used to guide the optimization of the reinforcement learning model; the driving task refers to the specific operation goal that the car needs to complete in a specific environment, including the parking method and target position; the task distance refers to the measure of the similarity of the impact of two parking actions on the completion of the driving task in the action space.

[0032] In the specific implementation, first, the driving task of the car during automatic parking can be obtained from the central control system of the car; secondly, all task influencing factors of the parking operation can be obtained from the driving task based on the reward shaping mechanism of reinforcement learning; then, the state-mapping model can be used to extract all parking actions of the parking operation from the action space, and the Euclidean distance of each group of adjacent parking actions can be used as the task distance between the corresponding adjacent parking actions, so as to obtain the task distance between each parking action; finally, the task influencing factor whose task distance is lower than the preset reward advance can be used as the reward factor. In the above way, all reward factors of the parking operation in the action space can be obtained.

[0033] It should be noted that the reward shaping mechanism is a method in reinforcement learning. By appropriately adjusting or guiding the environmental feedback rewards, the intelligent agent can learn effective strategies more quickly. This mechanism redesigns the reward structure so that the intelligent agent pays more attention to factors that directly contribute to the ultimate goal during task learning, thereby accelerating learning and optimizing the decision-making process. In the automatic parking system, the reward shaping mechanism helps the system extract key influencing factors (such as parking space size, obstacle location, surrounding environment, etc.) from complex driving tasks. By adjusting the reward feedback of influencing factors, the intelligent agent can strengthen its attention to factors such as parking position accuracy and environmental safety, thereby optimizing the decision-making process and improving the success rate and safety of automatic parking.

[0034] In some embodiments, determining the reward trajectory of the vehicle after automatic parking based on the instant rewards of each reward factor can be implemented by the following steps: Determine the instant reward for each reward factor in the parking maneuver; Filter out the concentrated value of rewards in parking operation from all instant rewards; Determine the reward volatility for parking maneuvers based on all instant rewards; A reward prediction is performed on the initial parking trajectory of the car according to the reward concentration value and the reward volatility to obtain a reward trajectory of the car after automatic parking.

[0035] It should be noted that, in the present application, the reward trajectory represents the parking trajectory after the decision-making action and environmental feedback during automatic parking; in specific implementation, firstly, for each reward factor in the parking operation, the distance reward, safety reward and time reward of the reward factors in the parking operation are obtained from the historical parking records, and the weighted average of the distance reward, safety reward and time reward can be used as the instant reward of the reward factor. The instant reward of each reward factor in the parking operation can be obtained in the above manner, and the instant reward reflects the effect of the current action in the specific environment; secondly, the mode of all instant rewards can be used as the reward concentration value in the parking operation, and the reward concentration value reflects the overall performance of the current parking operation. If the reward concentration value is high, it indicates that the overall performance of the parking operation is good, otherwise, Then, the operation strategy may need to be adjusted; then, the standard deviation of all instant rewards can be used as the reward volatility of the parking operation. The reward volatility reflects the stability of the parking operation. Lower volatility means that the operation process is more stable and reliable, while high volatility may indicate that the performance of the system varies greatly under different environmental conditions; finally, the ratio of reward volatility to reward concentration value can be used as the reward constraint of the car after automatic parking. The reward constraint represents the upper limit range set for the instant reward during the intelligent driving process. A weighted regression model based on a neural network is initialized, and the reward constraint is used as the regression coefficient of the weighted regression model. The weighted regression model is used to predict the reward for the initial parking trajectory of the car, and the predicted result can be used as the reward trajectory of the car after automatic parking.

[0036] In step 103, the state space of the sensor noise of the automobile during automatic parking is determined through the sensor noise information of the automobile during automatic parking, and then the parking control instruction is track-matched according to each characterization quality in the state space of the sensor noise and the driving decision in the action space to obtain the anti-interference performance of the parking operation of the automobile during automatic parking.

[0037] In some embodiments, the state space of the sensor noise of the vehicle during automatic parking is determined by using the sensor noise information of the vehicle during automatic parking, referring to Figure 3 The figure is a schematic diagram of a process for determining the state space of sensor noise in some embodiments of the present application. In this embodiment, determining the state space of sensor noise can be implemented by the following steps: In step 1031, sensor noise information and the state of the vehicle itself are obtained when the vehicle is in automatic parking; In step 1032, the sensor noise information and the self-state are combined based on state representation learning in reinforcement learning to generate a representation quality of each parking action of the automobile during automatic parking; In step 1033 , the state space of the sensor noise of the automobile during automatic parking is constructed according to all the characterization qualities.

[0038] It should be noted that in this application, state space is a multidimensional space used to describe the various states of sensor noise around the vehicle during automatic parking; characterization quality is an indicator that describes the effect of the parking action; sensor noise information refers to the physical environment around the vehicle, including all static and dynamic elements that may affect the vehicle's behavior; self-state refers to the vehicle's current dynamic characteristics, such as position, speed, acceleration, etc., which describe the vehicle's behavioral state.

[0039] In the specific implementation, first, the sensor noise information and the self-state during automatic parking can be obtained in the center console of the car; then, a state representation learning model based on reinforcement learning is initialized, and the sensor noise information is used as the external state feature of the state representation learning model, and the self-state is used as the internal state feature of the state representation learning model. The sensor noise information and the self-state are combined using the state representation learning model, and the combined result can be used as the characterization quality of each parking action of the car during automatic parking; finally, the characterization quality is reduced by using a dimensionality reduction method (for example: PCA principal component analysis) to generate a compact and representative state space, which can be used as the state space of the sensor noise of the car during automatic parking. The dimensionality reduction method can reduce the high-dimensional state space to an operational low-dimensional space, so that the center console of the car can effectively process and store it.

[0040] In some embodiments, the following steps may be used to match the trajectory of the parking control command according to each characterization quality in the state space of the sensor noise and the driving decision in the action space to obtain the anti-interference performance of the parking operation of the automobile during automatic parking: For each parking action during automatic parking, determining a control deviation of the parking operation according to a characterization quality corresponding to the parking action in the state space of the sensor noise; determining a decision degree of a parking action according to the driving decision in the action space; Based on the control deviation and the decision degree, spatial matching is performed on the driving trajectory of the vehicle after executing the parking control command to obtain the noise matching degree between the parking action and the sensor noise information, and then the noise matching degree between each parking action and the sensor noise information during automatic parking is obtained; The overall noise adaptation determines the vehicle's resistance to interference during automatic parking maneuvers.

[0041] It should be noted that, in this application, anti-interference performance refers to the ability of parking operation to maintain normal operation, stability and accuracy under different sensor noise information conditions; noise matching is an indicator to measure the degree of adaptation between parking action and sensor noise information; decision-making degree reflects the effectiveness index of parking action under specific environment; control deviation refers to the gap between expected state and actual state.

[0042] In specific implementation, first, for each parking action during automatic parking, the expected quality value of the parking action is obtained, and the difference between the expected quality value and the representation quality corresponding to the parking action in the state space of the sensor noise can be used as the control deviation of the parking operation; secondly, the decision utility function can be used to calculate the decision confidence index of the driving decision as the decision degree of the parking action; then, a spatial matching model is initialized, the decision degree is used as the matching parameter of the spatial matching model, and the control deviation is used as the correction parameter of the spatial matching model. The driving trajectory of the car after executing the parking control command is spatially matched using the spatial matching model, and the result of the spatial matching can be used as the noise matching degree between the parking action and the sensor noise information. The noise matching degree between each parking action and the sensor noise information during automatic parking can be obtained in the above manner; finally, the set of all noise matching degrees can be used as the value range of anti-interference, so as to obtain the anti-interference of the parking operation of the car during automatic parking.

[0043] In step 104, a deviation trajectory of the vehicle after automatic parking is determined according to the reward trajectory and the anti-interference performance, and adaptive constraint adjustment is performed on the decision-making action of the vehicle in automatic parking based on the deviation trajectory.

[0044] In some embodiments, determining the deviation trajectory of the vehicle after automatic parking according to the reward trajectory and the anti-interference performance can be implemented by the following steps: Determining a negative rate of a decision action according to the anti-interference performance; Determine a decision loss of a decision action by using the reward trajectory and the interference resistance; The reward trajectory is predicted to have a deviation according to the negative rate and the decision loss, so as to obtain a deviation trajectory of the car after automatic parking.

[0045] It should be noted that in this application, the deviation trajectory refers to the difference between the vehicle's expected trajectory and the expected trajectory during the automatic parking process; the decision loss represents the deviation between the decision action and the expected reward, and the decision loss measures the potential loss of the action; the negative rate represents the frequency with which the decision action produces adverse or negative events in the environment, and the negative rate quantifies the possibility of failure of the action.

[0046] In the specific implementation, first, the number of values ​​in the anti-interference property that are lower than the preset mercy matching threshold is counted as the number of unfavorable actions, and the number of all values ​​of the anti-interference property is taken as the total number of actions. The ratio of the number of unfavorable actions in the anti-interference property to the total number of actions can be used as the negative rate of the decision action; then, the absolute value of the difference between the mean of all values ​​in the anti-interference property and the reward constraint corresponding to the reward trajectory can be used as the decision loss of the decision action; finally, the product of the negative rate and the decision loss can be used as the decision risk of automatic parking, and the decision risk represents the potential risk of the decision action. The deviation trajectory reflects the safety of the action in a specific environment. A weighted regression model based on a neural network is initialized, and the decision risk is used as the regression coefficient of the weighted regression model. The weighted regression model is used to predict the deviation of the initial parking trajectory of the car, and the prediction result can be used as the deviation trajectory of the car after automatic parking.

[0047] In some embodiments, adaptive constraint adjustment of the decision-making action of the automobile in automatic parking based on the deviation trajectory can be achieved in the following manner, namely: if the deviation trajectory is not within the preset trajectory deviation range, the decision-making action in automatic parking is constrained and adjusted using the central console of the automobile until the deviation trajectory is within the preset trajectory deviation range; if the deviation trajectory is within the preset trajectory deviation range, no adjustment is made. It should be noted that in this application, the trajectory deviation range refers to the limit of acceptable trajectory deviation set for parking operations in the automatic driving system; in specific implementation, the trajectory deviation ranges of a large number of parking actions are obtained through experiments, and the 95% quantile of all deviation trajectories can be used as the trajectory deviation range.

[0048] In the present application, the deviation trajectory of the car after automatic parking is determined according to the reward trajectory and the anti-interference performance, and the decision-making action of the car in automatic parking is adaptively constrained and adjusted based on the deviation trajectory; firstly, in the automatic parking system, the reward trajectory can be determined to obtain the parking trajectory after the decision-making action and environmental feedback during automatic parking, thereby providing a benign feedback mechanism for the decision-making process of automatic parking. By real-time monitoring of the instant reward of each decision action, the car can evaluate its effectiveness and safety in a specific environment. The reward trajectory not only provides a quantitative standard for action selection, but also ensures that the car will not deviate from the safety when optimizing the decision. The full range ensures that when executing automatic parking, actions that can reduce adverse events (such as collisions, scratches, etc.) are given priority; then, the anti-interference performance is determined to obtain the ability of the parking operation to maintain normal operation, stability and accuracy under different sensor noise information conditions. By accurately matching the state space and decision-making actions of the sensor noise, the car can adjust the parking strategy in real time according to the actual environmental conditions, so that the vehicle can safely and effectively perform parking operations in various complex scenarios; in summary, based on the above scheme, the constraint adjustment of the automatic parking action of the car can be realized, thereby improving the environmental adaptability of the automatic parking of the car.

[0049] In addition, in another aspect of the present application, in some embodiments, the present application provides an intelligent driving decision system based on reinforcement learning, and the intelligent driving decision system based on reinforcement learning also includes a parking control unit, referring to Figure 4 , which is a schematic diagram of the structure of a parking control unit according to some embodiments of the present application, and the parking control unit includes: a monitoring module 201, a processing module 202 and an execution module 203, which are respectively described as follows: Monitoring module 201, in this application, monitoring module 201 is mainly used for real-time monitoring of the decision action of the automatic parking of the automobile when the automobile issues a parking control command for automatic parking in intelligent driving, and mapping the decision action into the action space of automatic parking based on the parking control command; Processing module 202, in the present application, the processing module 202 is used to extract all reward factors of the parking operation from the action space based on the reward shaping mechanism of reinforcement learning, and then determine the reward trajectory of the car after automatic parking based on the instant reward of each reward factor; It should be noted that the processing module 202 is also used to determine the state space of the sensor noise of the automobile during automatic parking through the sensor noise information of the automobile during automatic parking, and then perform trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space, so as to obtain the anti-interference performance of the parking operation of the automobile during automatic parking; The execution module 203 in the present application is mainly used to determine the deviation trajectory of the car after automatic parking based on the reward trajectory and the anti-interference performance, and to perform adaptive constraint adjustment on the decision-making action of the car in automatic parking based on the deviation trajectory.

[0050] The above describes in detail the examples of the intelligent driving decision-making system and method based on reinforcement learning provided by the embodiments of the present application. It can be understood that the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0051] In some embodiments, the present application also provides a computer device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned reinforcement learning-based automatic parking control method.

[0052] In some embodiments, reference Figure 5 , the dotted line in the figure indicates that the unit or the module is optional, and the figure is a structural diagram of a computer device for implementing an automatic parking control method based on reinforcement learning according to an embodiment of the present application. The automatic parking control method based on reinforcement learning described in the above embodiment can be Figure 5 The computer device shown in the figure is implemented, and the computer device includes at least one processor 301, a memory 302 and at least one communication unit 305. The computer device can be a terminal device, a server or a chip.

[0053] The processor 301 may be a general-purpose processor or a special-purpose processor. For example, the processor 301 may be a central processing unit (CPU), which may be used to control the computer device, execute software programs, and process data of the software programs. The computer device may also include a communication unit 305 to implement signal input (reception) and output (transmission).

[0054] For example, the computer device may be a chip, the communication unit 305 may be an input and / or output circuit of the chip, or the communication unit 305 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other devices.

[0055] For another example, the computer device may be a terminal device or a server, and the communication unit 305 may be a transceiver of the terminal device or the server, or the communication unit 305 may be a transceiver circuit of the terminal device or the server.

[0056] The computer device may include one or more memories 302, on which a program 304 is stored. The program 304 can be executed by the processor 301 to generate instructions 303, so that the processor 301 performs the method described in the above method embodiment according to the instructions 303. Optionally, data (such as a target audit model) can also be stored in the memory 302. Optionally, the processor 301 can also read the data stored in the memory 302, and the data can be stored at the same storage address as the program 304, or the data can be stored at a different storage address from the program 304.

[0057] The processor 301 and the memory 302 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0058] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 301. The processor 301 can be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0059] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0060] For example, in some embodiments, the present application also provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are executed on a computer, the computer implements the above-mentioned reinforcement learning-based automatic parking control method.

[0061] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0062] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An automatic parking control method based on reinforcement learning, characterized in that: The steps include: When the car issues a parking control command to perform automatic parking in intelligent driving, real-time monitoring of the car's automatic parking decision action is performed, and mapping the decision action into an automatic parking action space based on the parking control command; A reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, and then determines the reward trajectory of the car after automatic parking based on the immediate rewards of each reward factor; Determining the state space of the sensor noise of the vehicle during automatic parking by using the sensor noise information of the vehicle during automatic parking, and then performing trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space, so as to obtain the anti-interference performance of the parking operation of the vehicle during automatic parking; The deviation trajectory of the vehicle after automatic parking is determined according to the reward trajectory and the anti-interference performance, and the decision-making action of the vehicle in automatic parking is adaptively constrained and adjusted based on the deviation trajectory.

2. The method according to claim 1, characterized in that Mapping the decision action to an action space of automatic parking based on the parking control instruction specifically includes: Obtain parking control instructions for the car during automatic parking; Decomposing the decision action based on the parking control instruction to obtain all parking actions of automatic parking; The action space of automatic parking is constructed based on all parking actions.

3. The method according to claim 1, characterized in that The reward shaping mechanism based on reinforcement learning extracts all reward factors of the parking operation from the action space, including: Get the driving task of the car during automatic parking; A reinforcement learning-based reward shaping mechanism obtains all task influencing factors of the parking operation from the driving task; Extracting all parking actions of the parking operation from the action space, and then determining the task distance between each parking action; Based on all task distances, reward factors are screened out from all task influencing factors, thereby obtaining all reward factors for the parking operation in the action space.

4. The method according to claim 1, characterized in that The reward trajectory of the car after automatic parking is determined by the instant rewards of each reward factor, including: Determine the instant reward for each reward factor in the parking maneuver; Filter out the concentrated value of rewards in parking operation from all instant rewards; Determine the reward volatility for parking maneuvers based on all instant rewards; A reward prediction is performed on the initial parking trajectory of the car according to the reward concentration value and the reward volatility to obtain a reward trajectory of the car after automatic parking.

5. The method according to claim 1, characterized in that The state space of the sensor noise of the car during automatic parking is determined by the sensor noise information of the car during automatic parking, which specifically includes: Obtain sensor noise information and the vehicle's own status during automatic parking; The sensor noise information and the self-state are combined based on state representation learning in reinforcement learning to generate the representation quality of each parking action of the automobile during automatic parking; The state space of the sensor noise of the car during automatic parking is constructed based on all the characterization qualities.

6. The method according to claim 1, characterized in that Track matching of the parking control command is performed according to each characterization quality in the state space of the sensor noise and the driving decision in the action space to obtain the anti-interference performance of the parking operation of the automobile during automatic parking, specifically including: For each parking action during automatic parking, determining a control deviation of the parking operation according to a characterization quality corresponding to the parking action in the state space of the sensor noise; determining a decision degree of a parking action according to the driving decision in the action space; Based on the control deviation and the decision degree, spatial matching is performed on the driving trajectory of the vehicle after executing the parking control command to obtain the noise matching degree between the parking action and the sensor noise information, and then the noise matching degree between each parking action and the sensor noise information during automatic parking is obtained; The overall noise adaptation determines the vehicle's resistance to interference during automatic parking maneuvers.

7. The method according to claim 1, characterized in that Determining the deviation trajectory of the vehicle after automatic parking according to the reward trajectory and the anti-interference performance specifically includes: Determining a negative rate of a decision action according to the anti-interference performance; Determine a decision loss of a decision action by using the reward trajectory and the interference resistance; The reward trajectory is predicted to have a deviation according to the negative rate and the decision loss, so as to obtain a deviation trajectory of the car after automatic parking.

8. An intelligent driving decision system based on reinforcement learning, the intelligent driving decision system based on reinforcement learning includes a parking control unit, characterized in that: The parking control unit comprises: A monitoring module, for monitoring the decision action of the automatic parking of the automobile in real time when the automobile issues a parking control command for automatic parking in intelligent driving, and mapping the decision action into an action space of automatic parking based on the parking control command; A processing module, configured to extract all reward factors of the parking operation from the action space based on a reward shaping mechanism of reinforcement learning, and then determine a reward trajectory of the vehicle after automatic parking based on the instantaneous rewards of each reward factor; The processing module is also used to determine the state space of the sensor noise of the automobile during automatic parking through the sensor noise information of the automobile during automatic parking, and then perform trajectory matching on the parking control instruction according to each characterization quality in the state space of the sensor noise and the driving decision in the action space, so as to obtain the anti-interference performance of the parking operation of the automobile during automatic parking; An execution module is used to determine a deviation trajectory of the vehicle after automatic parking according to the reward trajectory and the anti-interference performance, and to perform adaptive constraint adjustment on the decision-making action of the vehicle in automatic parking based on the deviation trajectory.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the automatic parking control method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions or codes, and when the instructions or codes are executed on a computer, the computer implements the automatic parking control method based on reinforcement learning as described in any one of claims 1 to 7.

Images