Vehicle and unmanned aerial vehicle sensing integrated end-to-end automatic parking method and related device

Abstract

The invention discloses an end-to-end automatic parking method fusing vehicle and unmanned aerial vehicle perception and a related device. The method comprises the steps that parking area overlook RGB data obtained by an unmanned aerial vehicle and vehicle body surrounding obstacle data obtained by a vehicle-mounted sensor of an automatic driving vehicle are collected; the overhead view RGB data of the parking area and the obstacle data around the vehicle body are fused by using a convolutional neural network to obtain available parking spaces, a travelable area, obstacles and vehicle attitude information; and based on the obtained available parking space, the available driving area, the obstacle and the vehicle attitude information, utilizing a hybrid reinforcement learning strategy to plan and output a parking action. According to the invention, global high-level perception of the regional parking environment is realized, and automatic parking can be realized under the conditions of normal, complex and even extremely difficult obstacles and parking spaces.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of road traffic control and relates to an end-to-end automatic parking method fusing vehicle and unmanned aerial vehicle perception and a related device. BACKGROUND

[0002] Vehicle parking is an indispensable routine component in daily driving. It directly affects road efficiency, safety and user convenience. In recent years, automatic driving has attracted much attention due to its potential to change the transportation industry. Automatic driving technology enhances road safety, improves traffic flow and achieves better fuel economy, and also improves the intelligence of vehicles in daily driving tasks such as parking. After a large amount of offline training, an automatic driving vehicle can generate automatic parking actions according to the perception results of on-board sensors.

[0003] Existing automatic parking technologies mostly rely on on-board sensors to perform environment perception and path planning. However, due to the limited field of view of these sensors, automatic driving vehicles cannot obtain a comprehensive understanding of the surrounding environment, which severely limits their parking ability in complex scenarios. Existing researches use unmanned aerial vehicles as an auxiliary source of global high-level perception, providing over-the-horizon perception capability and achieving early planning. However, due to problems such as information transmission delay and system response inconsistency, these researches mostly focus on urban logistics and transportation applications. Therefore, there is still a lack of effective solutions to the problem of using unmanned aerial vehicles to assist in automatic parking. SUMMARY

[0004] The application aims to provide an end-to-end automatic parking method fusing vehicle and unmanned aerial vehicle perception and a related device, which solves the problem that existing technologies cannot effectively realize unmanned aerial vehicle assisted automatic parking.

[0005] To achieve the above-mentioned purpose, the application adopts the following technical solutions: The end-to-end automatic parking method fusing vehicle and unmanned aerial vehicle perception comprises: collecting parking area overhead RGB data obtained by an unmanned aerial vehicle and vehicle body surrounding obstacle data obtained by an on-board sensor of an automatic driving vehicle; fusing the parking area overhead RGB data and the vehicle body surrounding obstacle data by using a convolutional neural network to obtain available parking spaces, drivable areas, obstacles and vehicle attitude information; based on the obtained available parking spaces, drivable areas, obstacles and vehicle attitude information, using a hybrid reinforcement learning strategy to plan and output parking actions.

[0006] Further, the convolutional neural network fusion process comprises: processing the parking area overhead RGB image obtained by the unmanned aerial vehicle through a CNN encoder to extract semantic features; The vehicle body surrounding obstacle data acquired by the vehicle-mounted sensor is converted to an image coordinate system by a calibration transformation to construct a fusion state.

[0007] Further, the hybrid reinforcement learning strategy includes a Bezier curve strategy and a reinforcement learning strategy, and the action output by the reinforcement learning strategy and the action output by the Bezier strategy are mapped by a switching function to generate a hybrid action.

[0008] Further, the Bezier curve strategy is:

[0009] wherein, is a control point, is a normalized parameter, is a control point number, is the order of the Bezier curve; The expected reward of the reinforcement learning strategy is:

[0010] wherein, is the long-term expected total reward of the state under the strategy performing the action , is the expectation of the immediate reward and the next state , is a dynamic model of the environment, is an immediate reward function, is the instantaneous feedback obtained after performing the action in the state , is a discount factor, is the expected total reward of entering the state after performing , and then performing the action selected by the strategy , is the long-term expected total reward of the state under the strategy , is the expectation of the action output by the strategy . Further, the hybrid action is:

[0011]

[0012] wherein, is the final output hybrid control action, is a switching function, is the action generated by the reinforcement learning,​​​​​ is a policy network of reinforcement learning, is a network learnable parameter, is a system state at time t, is a Bézier curve generated action, is a hybrid policy, is a probability ratio.

[0013] Further, the multi-objective reward function of the hybrid reinforcement learning policy is:

[0014] wherein, is a parking success reward, is a collision penalty, is a pose reward, , , is a weight coefficient.

[0015] Further, the steering control signal of the parking action is:

[0016] wherein, is a steering control signal, 、 、 are proportional, integral, and derivative gains of the steering control respectively, is an angle deviation; The speed control signal of the parking action is:

[0017] wherein, is a speed control signal, 、 、 are proportional, integral, and derivative gains of the speed control respectively, is a speed deviation.

[0018] The end-to-end automatic parking system fusing vehicle and UAV perception includes: A data module for collecting parking area overhead RGB data obtained by a UAV and vehicle body surrounding obstacle data obtained by a vehicle-mounted sensor of an autonomous vehicle; A fusion module for fusing the parking area overhead RGB data and the vehicle body surrounding obstacle data by using a convolutional neural network to obtain available parking space, drivable area, obstacle, and vehicle pose information; A planning module is configured to plan and output parking actions based on the obtained available parking spaces, drivable areas, obstacles and vehicle pose information by using a hybrid reinforcement learning strategy.

[0019] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method when executing the computer program.

[0020] A computer-readable storage medium stores a computer program, and the computer program implements the steps of the method when executed by a processor.

[0021] Compared with the prior art, the present application has the following beneficial effects: The present application provides an end-to-end automatic parking method combining vehicle and UAV perception, which uses a UAV to obtain overhead RGB data of a parking area and uses a vehicle-mounted sensor of an autonomous vehicle to obtain obstacle data around the vehicle body. Then, a convolutional neural network is used to fuse the two kinds of data to obtain available parking spaces, drivable areas, obstacles and vehicle pose information. Finally, the obtained information is transmitted to a hybrid reinforcement learning strategy to plan and output parking actions. The present application combines a UAV and a vehicle-mounted sensor of an autonomous vehicle to realize global high-level perception of a regional parking environment. The vehicle-UAV end-to-end framework enables the autonomous vehicle to have global high-level perception capability in a complex parking environment, thereby ensuring the safety of automatic parking. In addition, the hybrid reinforcement learning strategy is used to plan and control the parking actions of the vehicle, which helps the autonomous vehicle to drive into the expected parking position under the visual guidance of the UAV, greatly shortens the parking time and improves the accuracy of parking. The end-to-end automatic parking method combining vehicle and UAV perception proposed in the present application is scientific and reasonable, fast and accurate, considers the time efficiency of the autonomous vehicle during parking, and reduces unnecessary detours. At the same time, the hybrid reinforcement learning strategy has a higher success rate than the typical method of directly using reinforcement learning, and can realize automatic parking under normal, complex and even extreme difficulty conditions of obstacles and parking spaces. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 The flowchart of the end-to-end automatic parking method combining vehicle and UAV perception of the present application.

[0024] Figure 2 Figure 1 is a VD-E2E framework diagram of the autonomous vehicle of the present application.

[0025] Figure 3 Figure 2 is a VD-E2E flowchart of the autonomous vehicle of the present application.

[0026] Figure 4 is a simulation result diagram of the autonomous vehicle parking in a vertical parking space in Example 1 of the present application, wherein Figure (a) is a parking process diagram of the autonomous vehicle, Figure (b) is a trajectory diagram of a normal parking space, Figure (c) is a trajectory diagram of a complex parking space, Figure (d) is a trajectory diagram of an extreme parking space, Figure (e) is a speed diagram, Figure (f) is a yaw angle diagram, and Figure (g) is an acceleration diagram.

[0027] Figure 5 is a simulation result diagram of the autonomous vehicle parking in a parallel parking space in Example 1 of the present application, wherein Figure (a) is a parking process diagram of the autonomous vehicle, Figure (b) is a trajectory diagram of a normal parking space, Figure (c) is a trajectory diagram of a complex parking space, Figure (d) is a trajectory diagram of an extreme parking space, Figure (e) is a speed diagram, Figure (f) is a yaw angle diagram, and Figure (g) is an acceleration diagram.

[0028] Figure 6 is a real-world experiment result diagram of the autonomous vehicle parking in a vertical parking space in Example 2 of the present application, wherein Figure (a) is a parking process diagram of the autonomous vehicle, Figure (b) is a trajectory diagram of a normal parking space, Figure (c) is a trajectory diagram of a complex parking space, Figure (d) is a trajectory diagram of an extreme parking space, Figure (e) is a speed diagram, Figure (f) is a yaw angle diagram, and Figure (g) is an acceleration diagram.

[0029] Figure 7 is a real-world experiment result diagram of the autonomous vehicle parking in a parallel parking space in Example 2 of the present application, wherein Figure (a) is a parking process diagram of the autonomous vehicle, Figure (b) is a trajectory diagram of a normal parking space, Figure (c) is a trajectory diagram of a complex parking space, Figure (d) is a trajectory diagram of an extreme parking space, Figure (e) is a speed diagram, Figure (f) is a yaw angle diagram, and Figure (g) is an acceleration diagram.

[0030] Figure 8 Figure 8 is a schematic diagram of a long-distance autonomous vehicle parking in Example 3 of the present application.

[0031] Figure 9 is a trajectory diagram of a long-distance autonomous vehicle parking in Example 3 of the present application, wherein Figure (a) is a free space diagram of a No. 1 parking area, and Figure (b) is a free space diagram of a No. 2 parking area.

[0032] Figure 10 Figure 10 is a schematic diagram of an end-to-end autonomous parking system structure of a preferred embodiment of the present application, which integrates vehicle and UAV perception.

[0033] Figure 11 Figure 11 is a schematic diagram of an electronic device structure of a preferred embodiment of the present application. DETAILED DESCRIPTION

[0034] The exemplary embodiments of this application are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of the embodiments of the application by a person of ordinary skill in the art, and should not be construed as a limitation on the scope of the application. Thus, persons of ordinary skill in the art will recognize that other changes and modifications can be made to the embodiments described herein, without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0035] Obviously, the described embodiments are part but not all of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of this application.

[0036] It should be noted that the terminal involved in the embodiments of the present application can include, but is not limited to, a mobile phone, a personal digital assistant (PDA), a wireless handheld device, a tablet computer, a personal computer (PC), an MP3 player, an MP4 player, a wearable device (for example, smart glasses, a smart watch, a smart bracelet, etc.), a smart home device, and the like.

[0037] In addition, the term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the associated objects before and after it.

[0038] The application will be further described in detail below with reference to the accompanying drawings: Referring to Figure 1 The application provides an end-to-end automatic parking method fusing vehicle and unmanned aerial vehicle perception, which specifically comprises the following steps: Step one: design a vehicle-unmanned aerial vehicle end-to-end (VD-E2E) framework.

[0039] To solve the multi-objective optimization problem, an end-to-end learning framework is designed to realize the mapping from environmental observation to optimal control action. Through reinforcement learning, the policy network learns to generate actions that minimize the cost function over time, while ensuring that kinematic constraints and safety constraints are met through interaction with the environment and reward function design.

[0040] As Figure 2As shown, the vehicle-drone end-to-end (VD-E2E) parking framework aims to make autonomous vehicles safely and accurately park to a designated parking spot by drone visual guidance. The architecture integrates high-level visual perception with low-level state estimation, supporting real-time parking decision making. The framework can handle two different types of sensor inputs: (1) the first type is the visual observation data of the drone, which extracts parking space, drivable area, obstacle, and autonomous vehicle pose information through a convolutional neural network (CNN); (2) the second type is the numerical data obtained by the autonomous vehicle's on-board sensors, including the current vehicle pose collected by an inertial measurement unit (IMU) and obstacle information collected by a lidar. Then, a modular action selection mechanism is introduced, which dynamically switches between bezier curve-based actions and reinforcement learning actions according to the feasibility of the current state-action pair. The hybrid actor-critic architecture uses the PPO algorithm for optimization. The actor network outputs a modulated action , while the critic estimates the state value function. The advantage-weighted loss function is used to guide policy updates, and the importance ratio is applied to ensure stability in the hybrid action space. Finally, the total reward function is designed to reflect multiple objectives and is decomposed into distance reward, direction reward, and collision penalty. To ensure safety, the collision penalty is given the highest weight, while the distance and direction terms encourage efficient and accurate parking. An entropy reward is also included to promote exploration during training. The VD-E2E process is shown in Figure 3 . Initially, the system is initialized with the PPO parameters and sensor inputs of the drone (top-down RGB image) and autonomous vehicle (IMU and lidar). The multi-sensor fusion module processes these inputs to extract drivable areas, obstacle maps, and target-related positions, and encodes them into the current state representation. Then the hybrid policy network decides whether to use a traditional bezier-based controller or a reinforcement learning-based policy to generate control commands. Once an action is executed, the reward is estimated. Then all the states, actions, rewards, and next-time states are used to update the PPO policy. This closed-loop process is repeated at each time step until the termination condition is met.

[0041] Step 2: Sensor fusion and state representation.

[0042] In this framework, the heterogeneous sensor inputs of the drone and the vehicle are jointly used as inputs to the hybrid policy network. The drone captures a top-down RGB image of the parking area, which is processed by a CNN encoder to extract semantic features. These features are used to identify drivable areas, obstacles, autonomous vehicle pose , and candidate parking spaces .

[0043] At the vehicle side, the on-board IMU sensor provides real-time ego state information ; meanwhile, the LiDAR sensor collects the surrounding obstacle information of the ego vehicle , especially the occluded obstacles from the UAV perspective.

[0044] To ensure spatial consistency, a calibration transformation is adopted to convert the ego vehicle pose based on IMU to the image coordinate system, i.e. . Subsequently, a fused state representation is constructed as the input of the hybrid policy network , which finally outputs the control action for parking decision .

[0045] Step 3: Hybrid policy network architecture.

[0046] 1. Bezier curve policy To ensure path smoothness and satisfy geometric constraints, a Bezier curve is adopted to generate feasible parking trajectories. An n-th order Bezier curve is defined as: (1) where is the current vehicle position, is the control point, is the target vehicle position, is the normalized parameter (value range ([0,1])), is the control point index, and is the order of the Bezier curve. This curve ensures that the ego vehicle can still operate reliably when the learned policy fails or unexpected situations occur.

[0047] 2. Reinforcement learning policy The parking process is modeled as a Markov Decision Process (MDP), represented by a triple , where is the ego vehicle state space, is the action space, and is the reward function estimated from the and at each time step. The goal is to learn a policy that maximizes the future reward , where is the discount factor. To obtain the optimal policy, reinforcement learning algorithms mainly describe the expected reward that can be obtained by following the policy after performing action in state : (2) in, In strategy Under, state Execute action The long-term expected total reward (action value). For instant rewards and the next state The expectations, among which It is a dynamic model of the environment (state transition probabilities). For the immediate reward function, in the state Execute action The instantaneous feedback obtained afterwards As a discount factor, To execute Entering the state afterward Then execute the strategy. Selected action Expected total reward In strategy Under, state The long-term expected total reward (state value). For strategy Output action The expectation, that is, based on the policy in the state The probability distribution of actions is calculated by weighting the value of all possible actions.

[0048] The reinforcement learning PPO algorithm is used to model the old policy parameters before and after the update. With new strategy parameters The change between them is defined as the importance ratio. The PPO loss function is expressed as: (3) in, for The estimated advantage value at time, For hyperparameters, This is a hyperparameter used in the PPO algorithm to limit the policy update magnitude, and is used to limit the deviation range of the probability ratio from 1.

[0049] 3. Hybrid Strategy Fusion Traditional reinforcement learning-based methods rely on repeated trial and error for exploration, resulting in high training costs and inevitably leading the agent into dangerous states. Rule-based models, once a trajectory is generated, require almost no further adjustments during execution. To improve exploration capabilities and training efficiency, reinforcement learning strategies are employed... With Bessel strategy Combined. The mixing process at different times can be achieved through a switching function. represents the function that maps the action output by the reinforcement learning policy to the action output by the Bezier policy . The hybrid action is computed as follows: (4) where is the final output hybrid control action, is the switching function, is the action generated by the reinforcement learning (PPO), is the policy network of PPO, is the network learnable parameter, is the system state at time t, is the action generated by the Bezier curve, is the hybrid policy, is the probability ratio.

[0050] During training, the agent takes the hybrid action to interact with the environment for exploration; during backpropagation, the PPO is updated based on the hybrid action . The value function and the Q-function are finally updated through Bellman backup: (5) where is the loss function of the value function , is the loss function of the function, is the value function, is the learnable weight of the value network, is the function, is the learnable weight of the network, is the expectation over the states sampled from the experience replay pool , is the expectation over the hybrid actions output by the hybrid policy , is the immediate reward, is the discount factor, is the expectation over the distribution of the next state after executing the hybrid policy , is the value function of the next state.

[0051] ​​​​​The hybrid architecture realizes smooth switching between traditional path planning and data-driven policy learning, improving the safety and learning efficiency of the parking process.

[0052] 4. Reward function The reward function is crucial for guiding the learning process of the agent, directly affecting the convergence speed and behavior quality. In the parking process of autonomous vehicles, the reward function needs to encourage the vehicle to gradually approach the designated parking space and ultimately achieve precise alignment, while prohibiting collision behavior. Therefore, a multi-objective reward function is designed as follows: (6) where, is the reward for successful parking, is the collision penalty, is the attitude reward, , , is the weight coefficient, used to balance the contribution of each part. Distance reward encourages the vehicle to approach the parking target, defined as: (7) where, is the maximum distance observed (normalized to the range [-1, 0]), , is the two-dimensional coordinate of the autonomous vehicle (AV) at the current time, , is the two-dimensional center coordinate of the ith target parking space. indicates that the autonomous vehicle is exactly located at the parking space, and the more negative the value, the farther the distance.

[0053] is to promote the correct final attitude of the vehicle in the parking space, the heading reward is defined as: (8) where, is the attitude angle of the vehicle relative to the direction of the parking space. The reward is normalized to the range [-1, 0], indicates complete alignment of the attitude. When a collision occurs, set the collision penalty to strictly limit the agent from entering an unsafe state.

[0054] Step four: Collaborative parking controller.

[0055] ​When the hybrid policy network outputs trajectory parameters (such as control points) or automatic driving vehicle real-time actuator instructions, the controller needs to ensure that these instructions can be accurately executed in the longitudinal and lateral directions. To ensure stability and practicality, a classic proportional-integral-derivative (PID) controller is used for the generated parameterized path (such as a Bezier curve).

[0056] Let the current pose of the automatic driving vehicle at time t be and the speed be . The nearest feasible point in front of the trajectory is selected as the reference point , and the reference speed is assigned to it . To quantify the path tracking effect of the automatic driving vehicle, the following two main control errors are designed: (9) (10) where and are the angle deviation and speed deviation, respectively, is the reference point angle, is the current pose angle, is the reference point speed, is the current pose speed.

[0057] Based on these two errors, the PID controller further generates steering and throttle / brake signals, expressed as follows: (11) (12) where 、 、 are the proportional, integral, and derivative gains of the steering control, respectively; 、 、 are the proportional, integral, and derivative gains of the speed control, respectively. To ensure compliance and safety of the actuator instructions, the final generated control signals and need to be limited within the physical range of the automatic driving vehicle actuator system.

[0058] When reinforcement learning mode is selected, the agent directly outputs the automatic driving vehicle actuator signals without the above conversion process, but still needs to comply with the saturation constraint.

[0059] The application will be further described in detail through specific embodiments: Example 1 This example is a simulation experiment, which evaluates both parallel and perpendicular parking spaces and covers three difficulty levels: ordinary, complex, and extreme. Each parking space configuration is tested 10 times independently, and the surrounding vehicles around the target parking space of the autonomous vehicle are randomly generated. These vehicles are not always perfectly aligned with their own parking space center, but are randomly distributed in pose while keeping them completely within the parking space. It is worth noting that the number and pose of these vehicles are different in each test. The simulation screenshots and the autonomous vehicle parking trajectory graph (20 Hz) are shown in Figures 4 and 5, respectively.

[0060] Figures 4(a) and 5(a) show an example of the autonomous vehicle parking process in the CARLA simulator. In Figures 4(b) to 4(d) and 5(b) to 5(d), the blue-green gradient rectangles represent the parking trajectory of the VD-E2E framework, and the results show that VD-E2E performs well in both perpendicular and parallel parking spaces. In the perpendicular parking space scenario, the autonomous vehicle begins to decelerate at about 50 steps, and the speed drops to 0 m / s at 150 steps, with a maximum speed of about 2.5 m / s (Figure 4(e)); the acceleration remains within the range of -3 to +3 m / s 2 (Figure 4(g)); the yaw angle smoothly transitions from about 0° to 90° at 60 steps (Figure 4(f)); and the total parking time is about 10 seconds. In the parallel parking space scenario, the trajectory is "S" shaped, consistent with human driving behavior. The autonomous vehicle reaches a peak speed of 1.5-2.3 m / s within 10-20 steps, and then decelerates to 0 m / s between 200-250 steps (Figure 5(e)); the acceleration remains between -3 and +3 m / s 2 (Figure 5(g)); the yaw angle first increases and then decreases, reaching a peak of about 52° at around 100-120 steps, and finally returning to 0°, indicating the completion of the forward-reverse operation to align the vehicle body (Figure 5(f)); and the total parking time is about 12 seconds.

[0061] Example 2 This example is a real vehicle test, using a Hongqi EQM-5 electric and autonomous vehicle, equipped with a series of onboard sensors, including multiple laser radars (1 main laser radar, 2 blind area laser radars) and a GPS antenna providing high-precision positioning. In addition, a 5G router is installed in the vehicle's trunk to enable wireless communication with the drone. A DJI Mini3 drone is used to collect overhead RGB images, providing high-level visual perception for the autonomous vehicle. It is worth noting again that the data exchange between the autonomous vehicle and the drone is achieved through a dedicated 5G link.

[0062] Similar to the simulation experiment, the difference of real scene setup is that traffic cones are used to define the parking space boundary to prevent actual collision during the test. The screenshots of real scene test and the parking trajectory of autonomous vehicle (100Hz) are shown in FIG. 6 and FIG. 7, respectively.

[0063] FIG. 6(a) and FIG. 7(a) show the parking process of the autonomous vehicle in the real scene test. The VD-E2E control can complete the vertical and parallel parking operation at one time under the conditions of ordinary, complex, and extreme. The trajectory is smooth and the curvature is continuous (FIG. 6(b) to FIG. 6(d), FIG. 7(b) to FIG. 7(d)). In the vertical parking space scene, the vehicle speed quickly rises and maintains between 0.8-1.3 m / s, and drops to 0 m / s within 1800-1950 steps (FIG. 6(e)); the yaw angle monotonically increases to 90°, and the convergence speed is the fastest in the extreme scene (FIG. 6(f)). In the parallel parking space scene, the speed is about 0.8-1.2 m / s, and drops to 0 m / s within 2700-3000 steps (FIG. 7(e)); the yaw angle first increases and then decreases, and reaches a peak value of 50-55° at about 1000-1300 steps, and finally returns to 0° (FIG. 7(f)).

[0064] Compared with the simulation experiment, the real scene test has more strict speed and acceleration limits, smaller fluctuations, and longer completion time, but the trend is consistent with the CARLA simulation. One-time parking is achieved in both experimental environments, and the trajectory is smooth, indicating that the VD-E2E framework has strong cross-scene generalization ability.

[0065] Embodiment 3: In a typical urban environment, an autonomous vehicle relies only on on-board sensors to detect idle parking spaces, and the perception ability is limited, which needs to traverse and scan each parking area one by one, which is time-consuming and inefficient. To solve this problem, this embodiment introduces a UAV-assisted long-distance parking strategy. After completing the driving task in the test site, the autonomous vehicle needs to return to one of the two candidate parking areas near the office building. Two candidate trajectories are defined as shown in FIG. 8. Figure 8

[0066] When the autonomous vehicle starts to return, the high-level perception ability of the UAV is used to evaluate the idle condition of the parking space in No. 1 parking area. If an idle parking space is detected, the autonomous vehicle travels along trajectory 1 (yellow) and parks in No. 1 area; otherwise, it directly travels along trajectory 2 (cyan) to No. 2 area to avoid unnecessary detours. Without UAV assistance, the autonomous vehicle defaults to trajectory 1 and can only detect the idle condition of the parking space after reaching each area. To quantitatively evaluate the advantages of UAV-assisted planning, the time efficiency of the two trajectories under different parking space idle conditions is compared, and the results are shown in FIG. 9.

[0067] ​When driving along the trajectory 1, the autonomous vehicle takes about 200 seconds to reach the No. 1 parking area, and takes an additional about 85 seconds to continue to the No. 2 area. As shown in FIG. 9, with the assistance of the UAV: when there is a free parking space in the No. 1 area, the autonomous vehicle can determine the final parking trajectory at 155 seconds earlier than the traditional method relying only on the on-board sensor ; when there is no free parking space in the No. 1 area, the UAV can help the autonomous vehicle to identify a free parking space in the No. 2 area, and complete the parking trajectory planning at 220 seconds earlier than the method without the assistance of the UAV. These results show that the UAV-assisted perception can significantly improve the planning efficiency, so that the autonomous vehicle can make a parking decision earlier and avoid unnecessary detours.

[0068] The present application also provides an end-to-end autonomous parking system fusing the perception of the vehicle and the UAV, as shown in FIG. 10, which comprises a data module, a fusion module and a planning module. Figure 10

[0069] The data module is used to collect the overhead RGB data of the parking area obtained by the UAV and the obstacle data around the vehicle body obtained by the on-board sensor of the autonomous vehicle. The fusion module is used to fuse the overhead RGB data of the parking area and the obstacle data around the vehicle body by using a convolutional neural network to obtain the available parking space, drivable area, obstacle and vehicle attitude information. The planning module is used to plan the parking action by using a hybrid reinforcement learning strategy based on the obtained available parking space, drivable area, obstacle and vehicle attitude information.

[0070] It can be understood that the end-to-end autonomous parking system fusing the perception of the vehicle and the UAV provided by the present application corresponds to the end-to-end autonomous parking method fusing the perception of the vehicle and the UAV provided by the foregoing embodiments, and the related technical features of the end-to-end autonomous parking system fusing the perception of the vehicle and the UAV can refer to the related technical features of the end-to-end autonomous parking method fusing the perception of the vehicle and the UAV, which will not be described here.

[0071] Another object of the present application is to provide an electronic device, as shown in FIG. 11, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the end-to-end autonomous parking method fusing the perception of the vehicle and the UAV. Figure 11 The end-to-end autonomous parking method fusing the perception of the vehicle and the UAV comprises the following steps:

[0072] ​​​collecting overhead RGB data of a parking area acquired by a UAV and obstacle data around a vehicle body acquired by a vehicle-mounted sensor of an autonomous vehicle; fusing the overhead RGB data of the parking area and the obstacle data around the vehicle body by using a convolutional neural network to obtain available parking spaces, drivable areas, obstacles and vehicle pose information; based on the obtained available parking spaces, drivable areas, obstacles and vehicle pose information, outputting a parking action by using a hybrid reinforcement learning strategy.

[0073] A fourth object of the present application is to provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the end-to-end automatic parking method fusing vehicle and UAV perception.

[0074] The end-to-end automatic parking method fusing vehicle and UAV perception comprises the following steps: collecting overhead RGB data of a parking area acquired by a UAV and obstacle data around a vehicle body acquired by a vehicle-mounted sensor of an autonomous vehicle; fusing the overhead RGB data of the parking area and the obstacle data around the vehicle body by using a convolutional neural network to obtain available parking spaces, drivable areas, obstacles and vehicle pose information; based on the obtained available parking spaces, drivable areas, obstacles and vehicle pose information, outputting a parking action by using a hybrid reinforcement learning strategy.

[0075] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer usable program code embodied therein.

[0076] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1means for performing the function specified by the block or blocks.

[0077] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0079] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. An end-to-end automatic parking method fusing vehicle and drone perception, characterized in that, The method comprises the following steps: collecting overhead RGB data of a parking area acquired by a UAV and obstacle data around a vehicle body acquired by a sensor of an autonomous vehicle; fusing the overhead RGB data of the parking area and the obstacle data around the vehicle body by using a convolutional neural network to obtain available parking spaces, drivable areas, obstacles and vehicle posture information; planning and outputting parking actions by using a hybrid reinforcement learning strategy based on the obtained available parking spaces, drivable areas, obstacles and vehicle posture information.

2. The end-to-end automatic parking method fusing vehicle and UAV perception according to claim 1, wherein, The fusion process of the convolutional neural network comprises the following steps: processing overhead RGB images of a parking area acquired by a UAV by using a CNN encoder to extract semantic features; converting obstacle data around a vehicle body acquired by a sensor to an image coordinate system by using a calibration transformation to construct a fusion state. 3.The end-to-end automatic parking method of fusing vehicle and UAV perception according to claim 1, wherein, The hybrid reinforcement learning strategy comprises a Bezier curve strategy and a reinforcement learning strategy, and a hybrid action is generated by mapping actions output by the reinforcement learning strategy and actions output by the Bezier strategy by using a switching function.

4. The end-to-end automatic parking method fusing vehicle and UAV perception according to claim 3, wherein, The Bezier curve strategy comprises the following steps: wherein, is a control point, is a normalization parameter, is a control point index, is an order of the Bezier curve; an expected reward of the reinforcement learning strategy is: in, In strategy Under, state Execute action The long-term expected total reward For instant rewards and the next state Expectations For a dynamic model of the environment, For the immediate reward function, in the state Execute action The instantaneous feedback obtained afterwards As a discount factor, To execute Entering the state afterward Then execute the strategy. Selected action Expected total reward In strategy Under, state The long-term expected total reward For strategy Output action The expectation.

5. The end-to-end automatic parking method fusing vehicle and drone perception according to claim 3, wherein, the hybrid action is: wherein, is the final output mixed control action, is a switching function, is the action generated by reinforcement learning, is the policy network of reinforcement learning, is the network learnable parameter, is the system state at time t, is the action generated by Bezier curve, is the mixed policy, is the probability ratio.

6. The end-to-end automatic parking method fusing vehicle and drone perception according to claim 1, wherein, a multi-objective reward function of the hybrid reinforcement learning strategy is: wherein, rewarded for a successful parking, penalized for a collision, rewarded for the pose, , , is a weight coefficient.

7. The end-to-end automatic parking method fusing vehicle and drone perception according to claim 1, wherein, a steering control signal of the parking action is: wherein is a steering control signal, 、 、 are proportional, integral, derivative gains of the steering control, respectively, is an angular deviation; a speed control signal of the parking action is: wherein, is a speed control signal, 、 、 are proportional, integral, derivative gains of the speed control, respectively, is a speed error.

8. An end-to-end automated parking system that fuses vehicle and drone perception, characterized in that, The method comprises the following steps: a data module is configured to collect overhead RGB data of a parking area acquired by a UAV and obstacle data around a vehicle body acquired by a sensor of an autonomous vehicle; a fusion module is configured to fuse the overhead RGB data of the parking area and the obstacle data around the vehicle body by using a convolutional neural network to obtain available parking spaces, drivable areas, obstacles and vehicle posture information; a planning module is configured to plan and output parking actions by using a hybrid reinforcement learning strategy based on the obtained available parking spaces, drivable areas, obstacles and vehicle posture information.

9. A terminal device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to realize the steps of the method in any one of claims 1-7.

Images