Cloud-based assisted automatic parking method, system, terminal and medium

Abstract

The application provides a kind of cloud-based auxiliary automatic parking method and system, record the data of surround view camera, ultrasonic radar, wheel speed meter and inertial navigation when vehicle automatic parking, and corresponding data is uploaded and cloud parking calculation is requested when parking fails;Cloud server obtains pose information according to dead reckoning of surround view camera, wheel speed meter and inertial navigation data;Target detection is carried out according to surround view camera data, and the size and position of obstacle are obtained, while the obstacle is detected according to ultrasonic radar data;The obstacles detected by the two are merged to obtain obstacle information;Based on surround view camera data, obtain panoramic surround view image, obtain parking space coordinate information combined with obstacle information, send obstacle information and parking space coordinate information to vehicle end;Parking trajectory is solved in parking local coordinate system, and parking trajectory information is sent to vehicle end;After information fusion and verification of vehicle end, execute parking trajectory, complete cloud-based auxiliary automatic parking.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving, in particular, to an automatic assisted parking method and system based on cloud assistance, a terminal and a medium. BACKGROUND

[0002] The automatic assisted parking (APA) system can complete the parking task automatically by perceiving the surrounding environment, controlling the acceleration and deceleration and the steering angle of the vehicle, reduce the burden of human drivers and reduce the occurrence of accidents such as scratching. Some commercial applications have been developed. Most of the current APA systems are limited by the hardware computing power of the vehicle. In order to ensure real-time performance, complex perception and planning algorithms cannot be used. Therefore, when facing difficult example scenarios such as small parking spaces and narrow passable areas, the vehicle may fail to detect the parking space or plan the path. The cloud server can provide cheap and convenient computing power for real-time perception and planning in the cloud, and then issue instructions to the vehicle for execution. However, uploading sensor data and issuing instructions through the cloud server are easily affected by network instability and delay, which may cause safety hazards. Therefore, how to combine cloud computing power and vehicle computing power is an effective solution to automatic assisted parking.

[0003] Through retrieval, it is found that the Chinese invention patent application with the publication number CN112802346A discloses an autonomous parking system and method based on cloud sharing and map fusion. Camera images are collected, a simultaneous localization and mapping (SLAM) is used to establish a parking route and a semantic map of a parking space, and trajectory information is classified to obtain a trajectory map. A target trajectory map is generated according to the selected trajectory map, wherein the target trajectory map is a new trajectory map generated by fusing the selected trajectory map. The trajectory map is stored locally and / or in the cloud. Autonomous parking parameters are set according to the target trajectory map. The repositioning is performed from coarse to fine according to the autonomous parking parameters. Although this method realizes the combination of cloud computing power and vehicle computing power, it needs to pre-establish a semantic map of the parking route and the parking space. Therefore, the autonomous parking function cannot be realized in parking lots where the semantic map has not been established, and the generalization is poor. SUMMARY

[0004] The present application provides an automatic assisted parking method, system, terminal and medium based on cloud assistance to solve the above problems in the prior art.

[0005] According to one aspect of the present application, an automatic assisted parking method based on cloud assistance is provided, comprising:

[0006] During the operation of the vehicle APA, the surround view camera data, ultrasonic radar data, wheel speed meter data and inertial navigation data during automatic parking of the vehicle are recorded. When the automatic parking fails, the recorded data are uploaded and the cloud parking calculation is requested.

[0007] The cloud server performs dead reckoning according to the surround-view camera data, the wheel speed meter data, and the inertial navigation data to obtain pose information of the vehicle in a motion process;

[0008] The cloud server performs 3D target detection according to the surround-view camera data to obtain sizes and positions of surrounding obstacles, and simultaneously detects the surrounding obstacles according to the ultrasonic radar data; merges the detected obstacles to obtain obstacle information; obtains a panoramic surround-view image based on the surround-view camera data, and obtains parking space coordinate information in combination with the obstacle information; and sends the obstacle information and the parking space coordinate information to the vehicle end;

[0009] The cloud server takes a center of a rear axle of the vehicle as a starting point, takes a position of a center of a rear axle of the vehicle corresponding to a target parking space as an ending point, solves a parking trajectory in a parking local coordinate system, and sends the parking trajectory information to the vehicle end;

[0010] The vehicle end obtains the obstacle information, the vehicle coordinate information, and the parking trajectory information sent by the cloud server, fuses and checks the same, and executes the parking trajectory through a vehicle APA to complete cloud-assisted automatic parking.

[0011] Preferably, the dead reckoning according to the surround-view camera data, the wheel speed meter data, and the inertial navigation data to obtain pose information of the vehicle in a motion process comprises:

[0012] taking an initial position of the vehicle as an original point of a map coordinate system; pre-processing and feature extraction are performed on the surround-view camera image to obtain feature points, a relative pose change between different time points of the camera is estimated through matching of the feature points between continuous frames of the image, and a Kalman filter or a factor graph optimization is applied to solve a current camera pose and a vehicle pose (x, y, yaw) through prior calibration data to store attitude information P = {(x1, y1, yaw1),..., (xN, yN, yawN)} of each time point.

[0013] Preferably, the 3D target detection according to the surround-view camera data to obtain sizes and positions of surrounding obstacles, and simultaneously detecting the surrounding obstacles according to the ultrasonic radar data; merging the detected obstacles to obtain obstacle information comprises:

[0014] a multi-view 3D target detection method is used to detect surrounding objects from the surround-view camera image to obtain bounding box information of each obstacle;

[0015] position information of the detected obstacle is calculated according to a measurement distance of the ultrasonic radar and an installation position and orientation of the ultrasonic radar;

[0016] Both obstacles are transformed into the map coordinate system through the current vehicle pose (x, y, yaw), the detected obstacles are merged, and the obtained bounding box information and position information are taken as the obstacle information.

[0017] Preferably, the panoramic surround view image is obtained based on the surround view camera data, and the parking space coordinate information is obtained in combination with the obstacle information, including:

[0018] The surround view camera data with pre-selected calibrated camera internal and external parameters is transformed into an aerial view and spliced to obtain a panoramic surround view image, including:

[0019] For each camera, the fisheye camera image image is de-warped through the pre-calibrated internal parameters to obtain an image image1; the image image1 is transformed through a perspective transformation algorithm using the fisheye camera external parameters to obtain an image image2 in the aerial view angle; all obtained images image2 are superimposed to obtain a panoramic surround view image in the aerial view angle;

[0020] The parking space is detected by deep learning, and the suitable parking space coordinate information is obtained in combination with the obstacle information, including:

[0021] The obtained panoramic surround view image is subjected to parking space detection using a deep learning algorithm to obtain four vertices point1-point4 of the parking space in the parking space coordinate system, the vertices point1-point4 are transformed into the map coordinate system containing the obstacle information through the vehicle pose (x, y, yaw) to obtain vertices point1-point4, and the center slot_center and the orientation slot_yaw of the parking space are calculated to obtain the parking space coordinate information.

[0022] Preferably, the parking trajectory is solved in the parking local coordinate system with the center of the rear axle of the vehicle as the starting point and the center of the rear axle of the vehicle corresponding to the target parking space as the end point, including:

[0023] A grid map M with appropriate size is established in the map coordinate system, and the positions overlapping with the detected obstacle information are set as occupied; the current vehicle position and orientation are taken as the starting state S, and the detected parking space center and orientation are taken as the ending state G; a collision-free path T from the starting state S to the ending state G is planned in the map M using a hybrid A* algorithm.

[0024] Preferably, the obstacle information, vehicle coordinate information, and parking trajectory information are obtained, fused, and verified, including:

[0025] It is determined whether each path point of the path T collides with any obstacle in the obstacle information, and if no collision occurs, the verification is passed and is handed over to the bottom controller for execution.

[0026] Preferably, the automatic parking fails, comprising:

[0027] Parking space detection failure or path planning failure, at which the vehicle stops moving.

[0028] According to another aspect of the present application, there is provided a cloud-based assisted automatic assisted parking system, characterized in that it comprises: an APA module and a vehicle-cloud fusion module arranged at the vehicle end, and a positioning module, a perception module and a planning module arranged at the cloud server; wherein:

[0029] The APA module is configured to record surround view camera data, ultrasonic radar data, wheel speed meter data and inertial navigation data during automatic parking of the vehicle, and upload the recorded data to the cloud server and request cloud parking calculation when the automatic parking fails; and execute the parking trajectory verified by the vehicle-cloud fusion module;

[0030] The vehicle-cloud fusion module is configured to obtain obstacle information, vehicle coordinate information and parking trajectory information sent by the cloud server, and after fusion and verification, send the verified parking trajectory information to the APA module;

[0031] The positioning module is configured to perform dead reckoning according to the surround view camera data, wheel speed meter data and inertial navigation data, and obtain the pose information of the vehicle during movement;

[0032] The perception module is configured to perform 3D target detection according to the surround view camera data to obtain the size and position of the surrounding obstacles, and simultaneously detect the surrounding obstacles according to the ultrasonic radar data; merge the detected obstacles to obtain obstacle information; obtain a panoramic surround view image based on the surround view camera data, and obtain parking space coordinate information in combination with the obstacle information, and send the obstacle information and the parking space coordinate information to the vehicle-cloud fusion module;

[0033] The planning module is configured to take the center of the rear axle of the vehicle as the starting point, take the position of the center of the rear axle of the vehicle corresponding to the target parking space as the end point, solve the parking trajectory in the parking local coordinate system, and send the parking trajectory information to the vehicle-cloud fusion module.

[0034] According to a third aspect of the present application, there is provided a computer terminal, comprising a vehicle-mounted terminal and a cloud computing platform terminal; wherein the vehicle-mounted terminal and the cloud computing platform terminal each comprise: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to execute the method of any one of the above aspects of the present application, or runs the system described in the above aspects of the present application.

[0035] According to a fourth aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, is adapted to perform the method of any one of the preceding aspects of the present application, or to operate the system of the preceding aspects of the present application.

[0036] As the above technical solutions are adopted, the present application has at least one of the following beneficial effects compared with the prior art:

[0037] The automatic assisted parking method, system, terminal and medium based on cloud assistance provided by the present application utilize cloud computing power to assist in solving the automatic parking task of a vehicle, upload vehicle end information, and solve in the cloud, so that a better automatic assisted parking algorithm is realized, and the success rate of the automatic assisted parking task is improved.

[0038] The automatic assisted parking method, system, terminal and medium based on cloud assistance provided by the present application fuse information of the vehicle end and the cloud, improve the accuracy of the information, and guarantee the safety of the parking task.

[0039] The automatic assisted parking method, system, terminal and medium based on cloud assistance provided by the present application request cloud assistance for parking calculation when the APA automatic parking fails, do not need to continuously communicate with the cloud computing platform, reduce the communication cost and delay influence, and improve the robustness of the system.

[0040] The automatic assisted parking method, system, terminal and medium based on cloud assistance provided by the present application do not need to make complex modifications to the vehicle end, can adapt to various different vehicle models, and have low added cost. BRIEF DESCRIPTION OF DRAWINGS

[0041] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0042] Figure 1 The figure is a work flow diagram of the automatic assisted parking method based on cloud assistance in an embodiment of the present application.

[0043] Figure 2 The figure is a work flow diagram of the automatic assisted parking method based on cloud assistance in a preferred embodiment of the present application.

[0044] Figure 3 The figure is a schematic diagram of the component modules of the automatic assisted parking system based on cloud assistance in an embodiment of the present application.

[0045] Figure 4 The figure is a work schematic diagram of the automatic assisted parking method and system based on cloud assistance in a specific application example of the present application. DETAILED DESCRIPTION

[0046] The following will make a detailed description of the embodiments of the present application: the embodiments are implemented on the premise of the technical solutions of the present application, and give detailed implementation modes and specific operation processes. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0047] The automatic assisted parking (APA) system is an important application of the automobile active collision avoidance system in low speed and complex urban environment, and is also a specific embodiment of the intelligentization of the automobile active collision avoidance system. Most of the current APA systems are limited by the hardware computing power of the vehicle, and when using cloud computing power, they are extremely susceptible to network instability and delay. In view of the above problems, an embodiment of the present application provides an automatic assisted parking method based on cloud assistance, which combines the vehicle-side computing power and the cloud-side computing power, improves the success rate of automatic assisted parking through cloud assistance, in view of the problems that the vehicle-side computing power affects the parking success rate and the cloud server is extremely susceptible to network instability and delay when uploading sensor data and issuing instructions.

[0048] Specifically, as shown in Figure 1 The automatic assisted parking method based on cloud assistance provided by the embodiment can include the following operations:

[0049] S1, during the running of the vehicle APA, record the surround view camera data, ultrasonic radar data, wheel speed meter data and inertial navigation data during automatic parking of the vehicle, and when automatic parking fails, upload the recorded data and request cloud parking calculation;

[0050] S2, the cloud server performs dead reckoning according to the surround view camera data, the wheel speed meter data and the inertial navigation data, and obtains the pose information of the vehicle during the movement;

[0051] S3, the cloud server performs 3D target detection according to the surround view camera data, obtains the size and position of the surrounding obstacles, and simultaneously detects the surrounding obstacles according to the ultrasonic radar data; merges the detected obstacles to obtain obstacle information; obtains a panoramic surround view image based on the surround view camera data, and obtains parking space coordinate information in combination with the obstacle information, and sends the obstacle information and the parking space coordinate information to the vehicle side;

[0052] S4, the cloud server takes the center of the rear axle of the vehicle as the starting point, takes the position of the center of the rear axle of the vehicle corresponding to the target parking space as the ending point, solves the parking trajectory in the parking local coordinate system, and sends the parking trajectory information to the vehicle side;

[0053] S5, the vehicle side obtains the obstacle information, the vehicle coordinate information and the parking trajectory information sent by the cloud server, and after fusion and verification, executes the parking trajectory through the vehicle APA to complete the cloud assisted automatic parking.

[0054] In some preferred embodiments, the above-mentioned S1, automatic parking failure, includes:

[0055] If parking space detection or route planning fails, the vehicle will stop moving.

[0056] In some preferred embodiments, the above-mentioned S2, which calculates dead reckoning based on surround-view camera data, wheel speedometer data, and inertial navigation data to obtain the vehicle's position and attitude information during movement, may further include the following operations:

[0057] The Visual Localization and Mapping (VSLAM) technique is employed, specifically including the following steps:

[0058] The initial position of the vehicle is taken as the origin of the map coordinate system. The images from the surround-view camera are preprocessed and feature extracted to obtain feature points. The relative pose changes of the camera at different time points are estimated by matching the feature points between consecutive frames of the image. The pose of the current camera is obtained by combining wheel speed sensor data and inertial navigation data and applying Kalman filtering or factor graph optimization. The pose information (x, y, yaw) of the vehicle is obtained by solving the pose information of the vehicle at each time point using pre-calibrated data. The pose information P = {(x1, y1, yaw1), ..., (xN, yN, yawN)} is stored.

[0059] In some preferred embodiments, S3 above, which involves performing 3D target detection based on surround-view camera data to obtain the size and position of surrounding obstacles, and simultaneously detecting surrounding obstacles based on ultrasonic radar data; merging the obstacles detected by both methods to obtain obstacle information, may further include the following operations:

[0060] A multi-view 3D object detection method is used to detect surrounding objects from panoramic camera images, and the bounding box information of each obstacle is obtained, including: center position, orientation and size information;

[0061] The location of the detected obstacle is calculated based on the measurement distance of the ultrasonic radar and the installation location and orientation of the ultrasonic radar; the obstacle can be regarded as a cylinder, the center of the cylinder is the location of the obstacle, and the radius can be selected as a small value.

[0062] Both types of obstacles are transformed into the map coordinate system using the current vehicle pose (x, y, yaw). The detected obstacles are merged, and the resulting bounding box information and position information are used as obstacle information, denoted as O.

[0063] In some preferred embodiments, the above-mentioned S3, which involves acquiring a panoramic surround view image based on surround view camera data and obtaining parking space coordinate information by combining obstacle information, may further include:

[0064] Transform the surround view camera data with pre-selected calibrated camera internal and external parameters into bird's eye view and stitch to get panoramic surround view image; specifically including the following steps:

[0065] For each camera, the fisheye camera image image is de-warped by the pre-calibrated internal parameters to obtain image image1; image1 is transformed by perspective transformation algorithm using the fisheye camera external parameters to obtain image image2 in bird's eye view; all obtained images image2 are superimposed to obtain panoramic surround view image in bird's eye view;

[0066] Detect the parking space by deep learning method, and obtain suitable parking space coordinate information combined with obstacle information; specifically including the following steps:

[0067] The panoramic surround view image obtained in the previous step is detected by deep learning algorithm (such as YOLO series, GCNParkingSlot, etc.) to obtain the 4 vertices point1~point4 of the parking space in the parking space coordinate system, and the vertices point1'~point4' are obtained by transforming the vehicle pose (x, y, yaw) to the map coordinate system containing obstacle information, and the center slot_center and the orientation slot_yaw of the parking space are calculated to obtain the parking space coordinate information.

[0068] In some preferred embodiments, S4, the vehicle rear axle center is taken as the starting point, and the vehicle rear axle center position corresponding to the target parking space is taken as the ending point, and the parking trajectory is solved in the parking local coordinate system, which can further include the following operations:

[0069] The smooth parking trajectory is solved in the parking local coordinate system by using the hybrid A* path optimization algorithm, specifically including the following steps:

[0070] The smooth parking trajectory is solved in the parking local coordinate system by using the hybrid A* path optimization algorithm, specifically including the following steps:

[0071] A grid map M with appropriate size is established in the map coordinate system, and the position overlapping with the obstacle information O detected in S3 is set as occupied; the current vehicle position and orientation are taken as the starting state S, and the detected parking space center and orientation are taken as the ending state G, and a collision-free path T from the starting point S to the ending point G is planned in the map M by using the hybrid A* algorithm.

[0072] In some preferred embodiments, S5, the obstacle information, parking space coordinate information and parking trajectory information are obtained and fused and verified, which can further include the following operations:

[0073] Calculate whether each path point of path T collides with any obstacle in obstacle information O. If no collision occurs, the check passes and is handed over to the underlying controller for execution.

[0074] One embodiment of the present invention provides an automatic assisted parking system based on cloud-based assistance.

[0075] Specifically, such as Figure 2 As shown, the cloud-based automated parking system provided in this embodiment may include: an APA module and a vehicle-to-cloud fusion module deployed on the vehicle, and a positioning module, a perception module, and a planning module deployed on a cloud server; wherein:

[0076] The APA module is used to record surround view camera data, ultrasonic radar data, wheel speed sensor data, and inertial navigation data during automatic parking. When automatic parking fails, it uploads the recorded data to the cloud server and requests cloud parking calculation; it also executes the parking trajectory verified by the vehicle-cloud fusion module.

[0077] The vehicle-cloud fusion module is used to obtain obstacle information, vehicle coordinate information and parking trajectory information sent by the cloud server, and after fusing and verifying them, send the verified parking trajectory information to the APA module.

[0078] The positioning module is used to perform dead reckoning based on surround-view camera data, wheel speedometer data, and inertial navigation data to obtain the vehicle's position and attitude information during its movement.

[0079] The perception module is used to perform 3D target detection based on surround view camera data, obtain the size and position of surrounding obstacles, and detect surrounding obstacles based on ultrasonic radar data; merge the obstacles detected by the two to obtain obstacle information; obtain a panoramic surround view image based on surround view camera data, and obtain parking space coordinate information by combining obstacle information, and send the obstacle information and parking space coordinate information to the vehicle-cloud fusion module.

[0080] The planning module is used to solve the parking trajectory in the local parking coordinate system, starting from the center of the rear axle of the vehicle and ending at the center of the rear axle of the vehicle corresponding to the target parking space, and then send the parking trajectory information to the vehicle-cloud fusion module.

[0081] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0082] The technical solution provided by the above embodiments of the present invention will be further described in detail below with reference to a specific application example.

[0083] As Figure 3 shown, and in combination Figure 2 with the specific application instance shown, the cloud-based assisted automatic parking system employed by the application provides, on the basis of the technical solutions provided by the above-mentioned embodiments of the application, includes:

[0084] - Vehicle-side module

[0085] 1. APA module:

[0086] Through the simple perception of the environment by the surround view camera and ultrasonic radar on the vehicle, the position of the obstacle and the parking space is determined, and the local positioning is performed using the wheel speed meter and inertial navigation, which is used to plan a collision-free path and control the vehicle to move into the parking space to complete the APA. If the parking space detection fails or the path planning fails, stop moving and upload the sensor data during the parking process to the cloud computing platform.

[0087] 2. Vehicle-cloud fusion module:

[0088] After the vehicle-side automatic parking fails, the cloud-based parking perception and planning results are obtained. The accurate obstacle information is obtained by fusing the obstacle perception information to prevent scratching and collision; the accurate parking space position is obtained by fusing the parking space perception information to improve the accuracy of parking; the safe, collision-free, and smooth trajectory is obtained by fusing the path planning information, and the trajectory is executed by the APA module.

[0089] - Cloud-side module

[0090] 1. Positioning module:

[0091] The starting point of the automatic parking is taken as the coordinate origin of the parking local coordinate system, and the wheel speed meter, inertial navigation, and surround view camera image are used to obtain the vehicle pose information during the parking process using the visual simultaneous localization and mapping (VSLAM) technology. A local parking coordinate system with the vehicle starting the APA function as the origin is established.

[0092] 2. Perception module:

[0093] The surround view image of the vehicle is obtained through the four surround view cameras, 3D target detection is performed in the surround view image to obtain the size and position of the surrounding obstacles, and ultrasonic detection is used to detect the obstacles around the vehicle. The obstacles detected by the two are merged. The camera image is converted to a bird's eye view perspective and spliced by using the pre-calibrated camera internal and external parameters, a panoramic surround view image is obtained, a deep learning algorithm is used to detect the parking space in the image, and the coordinates of the appropriate parking space are obtained in combination with the obstacle information. The obstacle information and the parking space information are transformed into the parking local coordinate system in combination with the vehicle pose.

[0094] 3. Planning module

[0095] The center of the rear axle of the vehicle is taken as the starting point, and the center of the rear axle corresponding to the parking space selected by the user is taken as the end point. A safe and smooth parking trajectory is solved in the parking local coordinate system using hybrid A* and path optimization algorithms, and the trajectory is sent to the vehicle. Under the support of the cloud computing platform, more complex methods can be used to calculate the parking trajectory in difficult scenarios.

[0096] Through the above-mentioned cloud-assisted automatic auxiliary parking system, the corresponding automatic auxiliary parking method is realized, as shown in Figure 4 , and as shown in Figure 3 , comprising the following steps:

[0097] S1, Figure 3 Among them, a vehicle that fails to perform an automatic auxiliary parking task uploads the surround view camera, ultrasonic radar, wheel speed meter, and inertial navigation data recorded during the parking process to the cloud computing platform after the APA failure, and requests cloud parking computing services.

[0098] S2, the cloud calculates the pose information during the vehicle motion according to the uploaded wheel speed meter, inertial navigation, and surround view camera data using VSLAM technology, and establishes a local parking map.

[0099] S3, 3D target detection is performed according to the image of the surround view camera, and the detection results of the ultrasonic radar are combined to detect two cars in the parking space as obstacles. By pre-calibrating the camera internal and external parameters, the camera image is processed into an aerial view and spliced to obtain a panoramic surround view image. Using a deep learning method, it can be detected that there are three parking spaces around, combined with the obstacle information, parking space 2 is selected as the parking space, and the obstacle information is sent to the vehicle.

[0100] S4, taking the center of the rear axle of the vehicle as the starting point and the center of the rear axle corresponding to the target parking space as the end point, a safe and smooth parking trajectory is solved using hybrid A* algorithm, and the trajectory is sent to the vehicle.

[0101] S5, the vehicle end acquires the parking perception and planning results of the cloud, fuses the obstacle perception information, parking space perception information, and path planning information. After fusion and verification of the information, the APA module executes the parking trajectory.

[0102] The automatic auxiliary parking method, system, terminal and medium based on cloud assistance provided by the above-mentioned embodiments of the application utilize cloud computing power to assist in solving the automatic parking task of a vehicle, upload vehicle-end information, solve in the cloud, realize an automatic auxiliary parking algorithm with better performance, and improve the success rate of the automatic auxiliary parking task; by fusing the information of the vehicle end and the cloud end, the accuracy of the information is improved, and the safety of the parking task is ensured; when the APA automatic parking fails, the cloud assistance parking calculation is requested, continuous communication with the cloud computing platform is not required, the communication cost and delay influence are reduced, and the robustness of the system is improved; without complex modification of the vehicle end, the system can adapt to various vehicle models, and the cost is low.

[0103] An embodiment of the application provides a computer terminal, including a vehicle terminal and a cloud computing platform terminal; the vehicle terminal and the cloud computing platform terminal both include a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor executes the computer program to execute the method in any one of the above-mentioned embodiments of the application or run the system in any one of the above-mentioned embodiments of the application.

[0104] Optionally, the memory is used for storing programs; the memory can include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviation: RAM), such as static random access memory (English: static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM) and the like; the memory can also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used for storing computer programs (such as application programs, functional modules and the like for realizing the above-mentioned method), computer instructions and the like, and the above-mentioned computer programs, computer instructions and the like can be stored in one or more memories in zones. And the above-mentioned computer programs, computer instructions, data and the like can be called by the processor.

[0105] The above-mentioned computer programs, computer instructions and the like can be stored in one or more memories in zones. And the above-mentioned computer programs, computer instructions, data and the like can be called by the processor.

[0106] The processor is configured to execute the computer program stored in the memory to implement each step in the method or each module of the system. Details can be referred to the description of the method and the system.

[0107] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled by a bus.

[0108] An embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method in any one of the above-mentioned embodiments of the present application or run the system in any one of the above-mentioned embodiments of the present application.

[0109] The above-mentioned embodiments of the present application are not exhaustive and are well known in the art.

[0110] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application.

Claims

1. A cloud-based automated assisted parking method, characterized in that, include: During the operation of the vehicle's APA, it records the surround-view camera data, ultrasonic radar data, wheel speed sensor data, and inertial navigation data when the vehicle is automatically parking. When automatic parking fails, it uploads the recorded data and requests parking calculations from the cloud. The cloud server performs dead reckoning based on the surround-view camera data, wheel speedometer data, and inertial navigation data to obtain the vehicle's position and attitude information during its movement; The cloud server performs 3D target detection based on the surround-view camera data to obtain the size and position of surrounding obstacles, and simultaneously detects surrounding obstacles based on the ultrasonic radar data; it merges the obstacles detected by both methods to obtain obstacle information; it acquires a panoramic surround-view image based on the surround-view camera data, and obtains parking space coordinate information by combining the obstacle information, and sends the obstacle information and the parking space coordinate information to the vehicle. The cloud server uses the rear axle center of the vehicle as the starting point and the rear axle center position of the target parking space as the ending point to solve the parking trajectory in the local parking coordinate system, and sends the parking trajectory information to the vehicle. After the vehicle receives obstacle information, vehicle coordinates, and parking trajectory information from the cloud server, and performs fusion and verification, it executes the parking trajectory through the vehicle's APA to complete cloud-assisted automatic parking.

2. The cloud-based automated parking assistance method according to claim 1, characterized in that, The step of performing dead reckoning based on the surround-view camera data, wheel speedometer data, and inertial navigation data to obtain the vehicle's position and attitude information during its movement includes: The initial position of the vehicle is taken as the origin of the map coordinate system. The images from the surround-view camera are preprocessed and feature extracted to obtain feature points. The relative pose changes of the camera at different time points are estimated by matching the feature points between consecutive frames of the image. The pose of the current camera is obtained by combining wheel speed sensor data and inertial navigation data and applying Kalman filtering or factor graph optimization. The pose information (x, y, yaw) of the vehicle is obtained by solving the pose information of the vehicle at each time point using pre-calibrated data. The pose information P = {(x1, y1, yaw1), ..., (xN, yN, yawN)} is stored.

3. The cloud-based automated parking assistance method according to claim 1, characterized in that, The step involves performing 3D target detection based on the surround-view camera data to obtain the size and position of surrounding obstacles, and simultaneously detecting surrounding obstacles based on the ultrasonic radar data. The obstacles detected by both are merged to obtain obstacle information, including: A multi-view 3D object detection method is used to detect surrounding objects from panoramic camera images to obtain bounding box information for each obstacle; The location information of the detected obstacle is calculated based on the measurement distance of the ultrasonic radar, as well as the installation location and orientation of the ultrasonic radar. Both types of obstacles are transformed into the map coordinate system using the current vehicle pose (x, y, yaw). The detected obstacles are merged, and the resulting bounding box information and position information are used as obstacle information.

4. The cloud-based automated parking assistance method according to claim 1, characterized in that, The step of acquiring a panoramic surround view image based on the surround view camera data and obtaining parking space coordinate information by combining the obstacle information includes: The panoramic camera data, with pre-calibrated camera intrinsic and extrinsic parameters, is transformed into a bird's-eye view and then stitched together to obtain a panoramic panoramic image, including: For each camera, the fisheye camera image is distorted using pre-calibrated intrinsic parameters to obtain image1; image1 is then transformed using the fisheye camera extrinsic parameters through a perspective transformation algorithm to obtain image2 from a bird's-eye view; all obtained images2 are then superimposed to obtain a panoramic view image from a bird's-eye view. Parking spaces are detected using deep learning, and combined with obstacle information, appropriate parking space coordinates are obtained, including: The obtained panoramic surround view image is used to detect parking spaces using a deep learning algorithm. The four vertices of the parking space, point1 to point4, are obtained in the parking space coordinate system. The vehicle pose (x, y, yaw) is transformed into the map coordinate system containing obstacle information to obtain the vertices point1' to point4'. The center of the parking space, slot_center, and the orientation, slot_yaw, are calculated to obtain the parking space coordinate information.

5. The cloud-based automated parking assistance method according to claim 1, characterized in that, The process of solving the parking trajectory in the local parking coordinate system, starting from the rear axle center of the vehicle and ending at the rear axle center of the target parking space, includes: Establish a grid map M of appropriate size in the map coordinate system, and set the positions that overlap with the detected obstacle information as occupied; take the current vehicle position and orientation as the starting state S, and the detected parking space center and orientation as the ending state G, and use the hybrid A* algorithm to plan a collision-free path T from the starting point S to the ending point G in the map M.

6. The cloud-based automated parking assistance method according to claim 1, characterized in that, Acquiring, fusing, and verifying the obstacle information, vehicle coordinate information, and parking trajectory information includes: Calculate whether each path point of path T collides with any obstacle in the obstacle information. If no collision occurs, the check passes and is handed over to the underlying controller for execution.

7. The cloud-based automated parking assistance method according to any one of claims 1-6, characterized in that, The automatic parking failure includes: If parking space detection or route planning fails, the vehicle will stop moving.

8. A cloud-based automated parking assistance system, characterized in that, include: The vehicle-mounted APA module and vehicle-to-cloud fusion module, as well as the positioning module, perception module, and planning module deployed on the cloud server; among which: The APA module is used to record surround view camera data, ultrasonic radar data, wheel speed sensor data, and inertial navigation data during automatic parking. When automatic parking fails, it uploads the recorded data to the cloud server and requests cloud parking calculation; and executes the parking trajectory verified by the vehicle-cloud fusion module. The vehicle-cloud fusion module is used to acquire obstacle information, vehicle coordinate information and parking trajectory information sent by the cloud server, fuse and verify them, and then send the verified parking trajectory information to the APA module. The positioning module is used to perform dead reckoning based on the surround-view camera data, wheel speedometer data, and inertial navigation data to obtain the vehicle's position and attitude information during its movement. The perception module is used to perform 3D target detection based on the surround-view camera data to obtain the size and position of surrounding obstacles, and simultaneously detect surrounding obstacles based on the ultrasonic radar data; merge the obstacles detected by the two methods to obtain obstacle information; acquire a panoramic surround-view image based on the surround-view camera data, and obtain parking space coordinate information by combining the obstacle information; and send the obstacle information and the parking space coordinate information to the vehicle-cloud fusion module. The planning module is used to solve the parking trajectory in the local parking coordinate system, starting from the rear axle center of the vehicle and ending at the rear axle center of the target parking space, and then send the parking trajectory information to the vehicle-cloud fusion module.

9. A computer terminal, comprising a vehicle-mounted terminal and a cloud computing platform terminal; wherein, Both the vehicle-mounted terminal and the cloud computing platform terminal include: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, can be used to execute the method of any one of claims 1-7, or to run the system of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-7, or to run the system of claim 8.

Images