Parking boundary identification method and system, terminal equipment and storage medium

Abstract

The invention provides a parking boundary recognition method and system, terminal equipment and a storage medium, and the method comprises the steps: determining a sample parking region picture of a to-be-parked vehicle, and obtaining a real environment boundary result and a real parking boundary result; obtaining a predicted environment boundary result and a predicted parking boundary result based on the initial parking boundary model and the sample parking area picture; obtaining an environment boundary loss function value based on the predicted environment boundary result and the real environment boundary result; obtaining a parking boundary loss function value based on the predicted parking boundary result and the real parking boundary result; obtaining a total boundary loss function value based on the environment boundary loss function value and the parking boundary loss function value; training the initial parking boundary model based on the total boundary loss function value to obtain a target parking boundary model; and inputting the current parking area picture of the to-be-parked vehicle into the target parking boundary model to obtain a target parking boundary. Therefore, the reliability of parking boundary identification is improved, and the safety of vehicle parking is improved.

Description

Technical Field

[0001] This invention relates to the field of parking technology, and in particular to a parking boundary recognition method, system, terminal device, and storage medium. Background Technology

[0002] Autonomous driving is gaining popularity, and safe autonomous driving often faces scenarios that require automatic parking.

[0003] Current approaches to parking for autonomous vehicles primarily focus on accurately navigating into parking spaces. However, it's clear that real-world parking environments are highly complex, often containing obstacles like walls and bollards, and frequently restricting the drivable area – aspects that current technologies haven't considered. Even with parking technologies designed for complex scenarios, such as deploying multiple independent dedicated models (one for drivable area segmentation and another for vehicle detection), significant computational redundancy arises. Multiple models repeatedly extract the same basic features, consuming extremely high computational resources and failing to meet real-time requirements. Furthermore, poor system coordination means the processing results from different models are difficult to perfectly align in time and space, potentially leading to inconsistent perception results. This ultimately makes the parking process unsafe. Summary of the Invention

[0004] The present invention aims to provide a parking boundary recognition method, system, terminal device and storage medium to solve the above-mentioned technical problems, improve the reliability of parking boundary recognition, and thus improve the safety of vehicle parking.

[0005] To address the aforementioned technical problems, this invention provides a parking boundary recognition method, comprising the following steps:

[0006] Determine the sample parking area image of the vehicle to be parked, and obtain the real environment boundary result and the real parking boundary result of the sample parking area image;

[0007] Based on the preset initial parking boundary model and the sample parking area image, the predicted environment boundary result and the predicted parking boundary result are obtained.

[0008] Based on the predicted environmental boundary results and the actual environmental boundary results, the environmental boundary loss function value is obtained; based on the predicted parking boundary results and the actual parking boundary results, the parking boundary loss function value is obtained; based on the environmental boundary loss function value and the parking boundary loss function value, the total boundary loss function value is obtained.

[0009] The initial parking boundary model is trained based on the total boundary loss function value to obtain the target parking boundary model;

[0010] Get an image of the current parking area for the vehicle to be parked;

[0011] The current parking area image is input into the target parking boundary model to obtain the target parking boundary.

[0012] In the above scheme, an environmental boundary loss function value is obtained by comparing the predicted environmental boundary result with the actual environmental boundary result. This environmental boundary loss function value improves the accuracy of identifying environmental objects or driving areas that need attention during parking. Similarly, a parking boundary loss function value is obtained by comparing the predicted parking boundary result with the actual parking boundary result, thus achieving accurate identification of non-grounded obstacles during parking. Specifically, based on the environmental boundary loss function value and the parking boundary loss function value, a total boundary loss function value is obtained. This total boundary loss function can simultaneously identify environmental object boundaries and non-grounded obstacle areas during parking. The initial parking boundary model is then trained based on the total boundary loss function to obtain a more realistic and comprehensive target parking boundary model, making the parking process safer and more reliable. The current parking area image is then input into the target parking boundary model to obtain a more accurate and comprehensive target parking boundary. Compared to existing technologies, this method does not require the computational resources of multiple models. In summary, this improves the reliability of parking boundary recognition, thereby enhancing vehicle parking safety.

[0013] Further, determining the sample parking area image of the vehicle to be parked, and obtaining the real environment boundary result and the real parking boundary result of the sample parking area image, includes:

[0014] Determine the location information of the vehicle to be parked and the sample parking area image;

[0015] Based on the preset boundary labeling model, the sample parking area image, and the location information, the real environment boundary result and the real parking boundary result are obtained.

[0016] In the above scheme, the image and location information are automatically processed by a preset boundary annotation model to generate real boundary results, which greatly reduces the workload and subjective error of manual annotation.

[0017] Furthermore, the real environment boundary results include drivable area boundary results. The process of obtaining the real environment boundary results and the real parking boundary results based on the preset boundary labeling model, the sample parking area image, and the location information includes:

[0018] Based on the location information, a drivable three-dimensional area is determined;

[0019] The outer contour is sampled in the drivable three-dimensional region to obtain the initial drivable three-dimensional annotation points;

[0020] Acquire the extrinsic and intrinsic parameters of the imaging device for the sample parking area image;

[0021] Based on the external parameter data of the shooting device and the initial 3D marking points of the drivable area, the 3D marking points of the target drivable area are obtained.

[0022] Based on the intrinsic parameter data of the shooting device and the three-dimensional annotation points of the target drivable area, the two-dimensional annotation points of the drivable area are obtained;

[0023] Connect the two-dimensional marker points of the drivable area to obtain the boundary result of the drivable area.

[0024] In the above scheme, the two-dimensional annotation range is determined by projection from three-dimensional space, and the manual annotation is transformed from full map search to fine-tuning within the defined area, avoiding unnecessary annotation work and further improving efficiency.

[0025] Furthermore, the real environment boundary results include temporary environmental object boundary results. The process of obtaining the real environment boundary results and the real parking boundary results based on the preset boundary labeling model, the sample parking area image, and the location information includes:

[0026] Based on the sample parking area image and the preset target temporary environmental object detection model, a two-dimensional bounding box for the temporary environmental object is obtained;

[0027] Based on the two-dimensional bounding box of the temporary environment object and the preset segmentation model of the target temporary environment object, the boundary result of the temporary environment object is obtained.

[0028] The above scheme adopts a process of detecting model localization and segmenting model initial labeling, which reduces the required manual resources and improves labeling efficiency.

[0029] Further, obtaining the real environment boundary result and the real parking boundary result based on the preset boundary annotation model, the sample parking area image, and the location information includes:

[0030] Obtain point cloud data of the parking area from the sample images;

[0031] Based on the point cloud data of the parking area, a three-dimensional bounding box of the parking area is obtained;

[0032] Based on the preset rotation and translation algorithm and the three-dimensional bounding box of the parking area, the corner points of the three-dimensional bounding box of the parking area are obtained;

[0033] Filter out the bottom bounding box corner points from the corner points of the three-dimensional bounding box of the parking area;

[0034] Based on the location information and the distance between the corner points of the bottom bounding box, the key parking boundary is selected from the three-dimensional bounding box of the parking area;

[0035] The corner points of the bottom bounding box corresponding to the parking key boundary are transformed to obtain the actual parking boundary result.

[0036] In the above scheme, the real parking boundary result of non-grounded parking is calculated by projecting the physical bottom surface of the three-dimensional bounding box of the parking area, which replaces the subjective estimation by manual estimation and ensures the authenticity of the real parking boundary result.

[0037] Further, the step of obtaining an environmental boundary loss function value based on the predicted environmental boundary result and the actual environmental boundary result; obtaining a parking boundary loss function value based on the predicted parking boundary result and the actual parking boundary result; and obtaining a total boundary loss function value based on the environmental boundary loss function value and the parking boundary loss function value, including:

[0038] Based on the predicted environmental boundary results, the actual environmental boundary results, and the preset environmental boundary classification loss algorithm, the environmental boundary classification loss value is obtained;

[0039] Obtain the intersection area and union area of ​​the predicted environmental boundary results and the actual environmental boundary results;

[0040] The environmental boundary intersection-union ratio loss value is obtained based on the ratio of the boundary intersection area to the boundary union area.

[0041] The environmental boundary loss function value is obtained by summing the environmental boundary classification loss value and the environmental boundary intersection-union ratio loss value.

[0042] In the above scheme, by combining the environmental boundary classification loss value and the environmental boundary intersection-union ratio loss value into a composite environmental boundary loss function, the model's classification accuracy at each pixel and the overall shape integrity of the predicted region are optimized simultaneously.

[0043] Further, the step of obtaining an environmental boundary loss function value based on the predicted environmental boundary result and the actual environmental boundary result; obtaining a parking boundary loss function value based on the predicted parking boundary result and the actual parking boundary result; and obtaining a total boundary loss function value based on the environmental boundary loss function value and the parking boundary loss function value, including:

[0044] Based on the predicted parking boundary results, the actual parking boundary results, and the preset environmental boundary classification loss algorithm, the parking boundary classification loss value is obtained;

[0045] Based on the distance between the predicted parking boundary result and the actual parking boundary result, the parking boundary distance loss value is obtained;

[0046] The parking boundary loss function value is obtained by summing the parking boundary classification loss value and the parking boundary distance loss value.

[0047] In the above scheme, by introducing a parking boundary distance loss value, the distance between the predicted parking boundary result and the actual parking boundary result in pixel coordinates is directly constrained, thereby driving the target parking boundary model trained by the loss function to output a boundary line that is highly accurate in geometric position, rather than just a semantically correct region.

[0048] Another embodiment of the present invention provides a parking boundary recognition system, comprising:

[0049] The determination module is used to determine the sample parking area image of the vehicle to be parked, and to obtain the real environment boundary result and the real parking boundary result of the sample parking area image;

[0050] The prediction module is used to obtain the predicted environment boundary result and the predicted parking boundary result based on the preset initial parking boundary model and the sample parking area image.

[0051] The function acquisition module is used to obtain the environmental boundary loss function value based on the predicted environmental boundary result and the actual environmental boundary result; to obtain the parking boundary loss function value based on the predicted parking boundary result and the actual parking boundary result; and to obtain the total boundary loss function value based on the environmental boundary loss function value and the parking boundary loss function value.

[0052] The training module is used to train the initial parking boundary model based on the total boundary loss function value to obtain the target parking boundary model;

[0053] The image acquisition module is used to acquire images of the current parking area of ​​the vehicle to be parked;

[0054] The boundary recognition module is used to input the current parking area image into the target parking boundary model to obtain the target parking boundary.

[0055] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of a parking boundary recognition method as described in the present invention.

[0056] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of a parking boundary recognition method of the present invention. Attached Figure Description

[0057] Figure 1 This is a schematic flowchart of a parking boundary recognition method according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of a parking boundary recognition system architecture provided in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Please see Figure 1 This embodiment provides a parking boundary recognition method, including:

[0061] Step S1: Determine the sample parking area image of the vehicle to be parked, and obtain the real environment boundary result and the real parking boundary result of the sample parking area image;

[0062] Step S2: Based on the preset initial parking boundary model and sample parking area images, obtain the predicted environment boundary results and the predicted parking boundary results;

[0063] Step S3: Based on the predicted environmental boundary results and the actual environmental boundary results, obtain the environmental boundary loss function value; based on the predicted parking boundary results and the actual parking boundary results, obtain the parking boundary loss function value; based on the environmental boundary loss function value and the parking boundary loss function value, obtain the total boundary loss function value.

[0064] Step S4: Train the initial parking boundary model based on the total boundary loss function value to obtain the target parking boundary model;

[0065] Step S5: Obtain an image of the current parking area for the vehicle to be parked;

[0066] Step S6: Input the current parking area image into the target parking boundary model to obtain the target parking boundary.

[0067] In the above scheme, the environmental boundary loss function value is obtained by comparing the predicted environmental boundary results with the actual environmental boundary results. This environmental boundary loss function value improves the accuracy of identifying environmental objects or driving areas of interest during parking. Similarly, the parking boundary loss function value is obtained by comparing the predicted parking boundary results with the actual parking boundary results, thus achieving accurate identification of non-grounded obstacles. Specifically, based on the environmental boundary loss function value and the parking boundary loss function value, a total boundary loss function value is obtained. This total boundary loss function enables the identification of both environmental object boundaries and non-grounded obstacles during parking. The initial parking boundary model is then trained based on the total boundary loss function to obtain a more realistic and comprehensive target parking boundary model, making the parking process safer and more reliable. The current parking area image is then input into the target parking boundary model to obtain a more accurate and comprehensive target parking boundary. Compared to existing technologies, this method does not require the computational resources of multiple models. In summary, this improves the reliability of parking boundary recognition, thereby enhancing vehicle parking safety.

[0068] In another embodiment, a sample parking area image of the vehicle to be parked is determined, and the true environmental boundary result and the true parking boundary result of the sample parking area image are obtained, including:

[0069] Determine the location information of the vehicles to be parked and obtain sample parking area images;

[0070] Based on the preset boundary labeling model, sample parking area images, and location information, the results of the real environment boundary and the real parking boundary are obtained.

[0071] It should be noted that there are several things to pay attention to in the sample parking area image. Based on the preset boundary labeling model, the sample parking area image and location information, after obtaining the real environment boundary result and the real parking boundary result, the scene in the sample parking area image can be roughly classified into the real environment boundary result and the real parking boundary result. The real environment boundary result refers to the ground objects that need to be paid attention to during parking, including: (1) the boundary result of the drivable area, which refers to the boundary of the drivable road section; (2) the boundary result of the fixed environmental object, which refers to the boundary of relatively large obstacles with fixed structures, such as: walls, pillars, curbs, green belts, etc.; (3) the boundary result of the temporary environmental object, which refers to the boundary of messy and scattered small obstacles, such as: cones, crash barriers, triangular warning signs, no parking signs, round stone blocks, column stone blocks, iron ground stakes, plastic ground stakes, and ground locks. The true parking boundary result refers to non-grounded objects that need to be considered during the parking process, such as other parked vehicles. The objects that affect the parking vehicle are not the grounded tires, but the non-grounded vehicle body, such as the door.

[0072] In another embodiment, the real environment boundary results include drivable area boundary results. Based on a preset boundary labeling model, sample parking area images, and location information, the real environment boundary results and real parking boundary results are obtained, including:

[0073] Based on location information, determine the drivable three-dimensional area;

[0074] The outer contour of the drivable 3D region is annotated to obtain the initial 3D annotation points of the drivable region.

[0075] Acquire the extrinsic and intrinsic parameters of the imaging device for the sample parking area images;

[0076] Based on the external parameter data of the shooting device and the initial 3D annotation points of the drivable area, the 3D annotation points of the target drivable area are obtained;

[0077] Based on the intrinsic data of the shooting device and the three-dimensional annotation points of the target drivable area, the two-dimensional annotation points of the drivable area are obtained;

[0078] Connect the two-dimensional markers of the drivable area to obtain the boundary result of the drivable area.

[0079] It should be noted that, to obtain the boundary results of the drivable area, sample parking area images are first acquired. The number of sample parking area images is unlimited. During the acquisition of the drivable area boundary results, the sample parking area images include multiple fisheye images, covering the front, rear, left, and right sides of the vehicle to be parked. Simultaneously, the extrinsic and intrinsic parameters of the imaging device used to capture the sample parking area images are acquired. The extrinsic parameters typically refer to camera extrinsic parameters, including focal length and distortion coefficients. The intrinsic parameters typically refer to camera intrinsic parameters, including the transformation matrix from the imaging device (camera) to the vehicle coordinate system.

[0080] Then, based on the location information of the vehicle waiting to park, a drivable 3D area is determined. For example, based on the location information, a circle with a radius of 13 meters is set as the drivable 3D area, centered on the vehicle (the vehicle waiting to park). This range covers most of the near-field perception area required for the parking scenario. The outer contour of the drivable 3D area is then annotated. For example, a point is marked on the circle every 1°, with each point corresponding to an angle, resulting in initial 3D annotation points for the drivable area. In this embodiment, 361 points are obtained, each initial drivable area 3D annotation point θ. i=i°, i = 0, 1, ..., 360. It's important to note that the height z of the drivable 3D region is set to 0 meters (ground level), because the drivable region boundary results only concern the drivable area on the ground. At this point, the coordinates of the initial drivable region 3D annotation points can be constructed. For example, if there are 361 initial drivable region 3D annotation points, the coordinates (x, y, z) of the point 13 meters ahead are (13, 0, 0). The calculation formula for other points is:

[0081]

[0082] Among them, P i P represents the coordinates of the initial drivable area 3D marker, i is the index of the initial drivable area 3D marker, and z is the height of the initial drivable area 3D marker, set to 0. 3D ={P1,P2,…,P i ,…,P n}, where n is the number of three-dimensional markers in the initial drivable area. In the above embodiment, n = 361.

[0083] Then, based on the extrinsic data of the imaging device and the initial 3D annotation points of the drivable area, the 3D annotation points of the target drivable area are obtained. Specifically: the rotation matrix T is obtained based on the extrinsic data of the imaging device. c Translation matrix R c This allows us to obtain the 3D labeled point P of the target drivable area in camera coordinates. cam =(X cam ,Y cam Z cam ), where X cam ,Y cam Z cam The xyz coordinates of the three-dimensional annotation points of the target's drivable area, P cam The calculation formula is:

[0084]

[0085] in, Represented as a translation matrix R c The transpose of the matrix, as we understand it, is also used to characterize the transformation based on the rotation matrix T. c Translation matrix R c For P 3D A rigid transformation is achieved by rotation and translation, so that P 3D As a rigid body (from the vehicle body to the camera) moves and rotates together, P is obtained. cam .

[0086] Then, based on the intrinsic data of the imaging device and the 3D annotation points of the target drivable area, the 2D annotation points of the drivable area are obtained. Specifically: the intrinsic data of the imaging device uses the Kannala-Brandt model, so the 3D annotation points of the target drivable area in the camera coordinate system can be projected onto the image coordinate system through fisheye projection to obtain the 2D annotation points (u,v) of the drivable area in the image. The calculation formula is as follows:

[0087] φ=arctan2(Y cam ,X cam );

[0088] r d =θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9 ;

[0089] u = f u ·r d ·cosφ+c u ;

[0090] v = f v ·r d ·sinφ+c v ;

[0091] Where k1, k2, k3, and k4 are the first, second, third, and fourth distortion parameters of the imaging device's intrinsic data, respectively, and f u ,f v c represents the focal length of the imaging device on the x and y axes. u ,c v Let θ be the principal point of the imaging device on the x and y axes. θ is the angle of incidence, i.e., the angle between the ray emanating from the optical center of the camera (imaging device) and the optical axis, in radians; φ is the projection of this point onto the XY plane of the camera coordinate system; r d The radius of the imaging point is the Euclidean distance from the principal point on the image plane.

[0092] The final two-dimensional label points (u,v) of the drivable area are used as the labeling range. Connecting the two-dimensional label points of the drivable area yields the boundary result of the drivable area, which is also used as the labeling range. Outside the labeling range, random labeling can be used to reduce the amount of labeling.

[0093] Furthermore, the real-world boundary results also include fixed obstacle boundary results. Specifically, similar to the process of obtaining the drivable area boundary results, to obtain the fixed obstacle boundary results, sample parking area images are first acquired. The number of sample parking area images is unlimited. In the process of obtaining the drivable area boundary results, the sample parking area images include multiple fisheye images, respectively including the front, rear, left, and right sides of the vehicle to be parked. Simultaneously, the extrinsic and intrinsic parameters of the imaging device that captured the sample parking area images are acquired. The extrinsic parameters typically refer to camera extrinsic parameters, including focal length and distortion coefficients. The intrinsic parameters typically refer to camera intrinsic parameters, including the transformation matrix from the imaging device (camera) to the vehicle coordinate system. Then, based on the location information of the vehicle waiting to park, a three-dimensional region of fixed obstacles is determined. For example, using the vehicle (the vehicle waiting to park) as the baseline, a boundary of 13 meters in front, behind, left, and right, and 3 meters in height is set as the three-dimensional region of fixed obstacles. This three-dimensional region of fixed obstacles is then labeled. For example, only targets within 13 meters are labeled, and this is extended upwards by 3 meters. Compared to the 361 points with z=0 above, here P... 3D There are also 361 additional points with z=3. The rest of the projection calculation process is the same as above. Finally, the fixed obstacle boundary result is obtained.

[0094] In another embodiment, the real environment boundary results include temporary environmental object boundary results. Based on a preset boundary labeling model, sample parking area images, and location information, the real environment boundary results and real parking boundary results are obtained, including:

[0095] Based on sample parking area images and a pre-set target temporary obstacle detection model, a two-dimensional bounding box for temporary environmental objects is obtained.

[0096] Based on the two-dimensional bounding box of the temporary environment and the preset target temporary obstacle segmentation model, the boundary results of the temporary environment are obtained.

[0097] It should be noted that the sample parking area images include many scene images containing traffic cones, crash barriers, warning triangles, no-parking signs, round bollards, columnar bollards, iron bollards, plastic bollards, and locked parking locks. A batch of existing sample parking area images were selected, and each temporary obstacle was meticulously annotated manually to draw its precise outline (segmentation annotation). This annotated data was used to train an initial temporary obstacle detection model, resulting in a target temporary obstacle detection model. This target temporary obstacle detection model can identify obstacles in the image and outline them using 2D bounding boxes. When new sample parking area images arrive, it's no longer necessary to manually redraw the outlines; the trained target temporary obstacle detection model automatically outlines the obstacles in the new sample parking area images, obtaining 2D bounding boxes for temporary environmental objects. Then, using the already trained target temporary obstacle segmentation model, the outlines of the obstacles are generated within each obtained 2D bounding box of the temporary environmental object. These serve as the boundary results for the temporary environmental objects. Furthermore, the outline of the temporary environment boundary result may not be perfect, but at this point, the relevant personnel only need to check and correct these automatically generated outlines, such as erasing the excess, filling in the missing parts, and trimming the edges, instead of drawing it from scratch. This is extremely efficient, and a fine outline subdivision of an image can be completed in just a few minutes.

[0098] In another embodiment, based on a preset boundary annotation model, sample parking area images, and location information, the real environment boundary results and the real parking boundary results are obtained, including:

[0099] Obtain point cloud data of the parking area from the sample images;

[0100] Based on the point cloud data of the parking area, the 3D bounding box of the parking area is obtained;

[0101] Based on the preset rotation and translation algorithm and the 3D bounding box of the parking area, the corner points of the 3D bounding box of the parking area are obtained;

[0102] Filter out the corner points of the bottom bounding box from the corner points of the 3D bounding box of the parking area;

[0103] Based on the location information and the distance between the corner points of the bottom bounding box, the key parking boundary is selected from the 3D bounding box of the parking area.

[0104] The corner points of the bottom bounding box corresponding to the key parking boundary are transformed to obtain the actual parking boundary result.

[0105] It should be noted that, similar to the process of obtaining the drivable area boundary results, to obtain the true parking boundary results, sample parking area images are first acquired. The number of sample parking area images is unlimited. In the process of obtaining the drivable area boundary results, the sample parking area images include multiple fisheye images, covering the front, rear, left, and right sides of the vehicle to be parked. Simultaneously, the extrinsic and intrinsic parameters of the camera device that captured the sample parking area images are acquired. The extrinsic parameters typically refer to camera extrinsic parameters, including focal length and distortion coefficients, while the intrinsic parameters typically refer to camera intrinsic parameters, including the transformation matrix from the camera to the vehicle coordinate system. In addition, the parking area point cloud data of the vehicles in this region of the sample images needs to be obtained using a 3Dbox detection device. In the parking area point cloud data, a 3D bounding box is labeled for each non-grounded target, such as a vehicle. The 3D bounding box includes the target's center position (x, y, z), dimensions (length l, frame w, height h), and orientation angle (yaw). The 3D bounding box of the parking area has 8 corner points. From these, we select the 4 bottom bounding box corner points, specifically:

[0106] (1) Define half length / half width / half height in the local coordinate system: dx = l / 2, dy = w / 2, dz = h / 2.

[0107] (2) Select 4 local corners from the 8 corner points of the object's local coordinate system (with the center as the origin and the axes aligned):

[0108] local corners=[±dx,±dy,-dz];

[0109] (3) Rotate the bottom bounding box corners around the Z-axis by the yaw angle to ensure they are facing the correct direction:

[0110] Rotation matrix R yaw for:

[0111]

[0112] Transform each local vertex Plocal from the object's local coordinate system to the lidar coordinate system (global coordinate system):

[0113] p lidar =R yaw ·p local +c;

[0114] Where, p local Let c = [x, y, z], where c is the eight corner points of the 3D bounding box of the parking area in the object's local coordinate system. T It is the center of the 3D bounding box of the parking area, p lidarThe eight corner points of the 3D bounding box of the parking area in the lidar coordinate system (global coordinate system): Select the four points with the smallest Z coordinate from the eight corners to obtain the four corner points of the bottom bounding box in the global coordinate system of the bottom quadrilateral.

[0115] Based on the corner points P1, P2, P3, and P4 of the bottom bounding box, four edges forming the bottom surface are P1→P2, P2→P3, P3→P4, and P4→P1. The Euclidean distance between the midpoint of each edge and the origin (0,0,0) of the vehicle (the vehicle waiting to park) is calculated. The two edges with the shortest distance are selected as the critical parking boundaries. This is because the parking scenario is most concerned with the boundary portion closest to the vehicle and at risk of collision. Then, using the intrinsic and extrinsic parameters of the imaging device (i.e., camera intrinsic parameter K and extrinsic parameter R / T) and other imaging device calibration parameters, including distortion D, the corner points of the bottom bounding box corresponding to the critical parking boundaries are transformed from the LiDAR coordinate system → camera coordinate system → image coordinate system. Coordinate transformation: LiDAR coordinate system → camera coordinate system, where T... lc and R lc It is the rotation and translation matrix (obtainable through calibration) in the extrinsic parameters of the lidar to the imaging device (camera).

[0116] P cam =R lc ·P lidar +T lc ;

[0117] Among them, P lidar It is the corner point of the bottom bounding box on the critical boundary of parking, P cam These are the corner points of the bottom bounding box in the transformed camera coordinate system.

[0118] The calculation process for projecting the image coordinate system using a fisheye or pinhole model is the same as before. The final result is the set of points projected onto the image, which represents the true parking boundary.

[0119] In another embodiment, an environmental boundary loss function value is obtained based on the predicted environmental boundary results and the actual environmental boundary results; a parking boundary loss function value is obtained based on the predicted parking boundary results and the actual parking boundary results; and a total boundary loss function value is obtained based on the environmental boundary loss function value and the parking boundary loss function value, including:

[0120] Based on the predicted environmental boundary results, the actual environmental boundary results, and the preset environmental boundary classification loss algorithm, the environmental boundary classification loss value is obtained;

[0121] Obtain the intersection area and union area of ​​the predicted environmental boundary results and the actual environmental boundary results;

[0122] The environmental boundary intersection-union ratio loss value is obtained based on the ratio of the boundary intersection area to the boundary union area.

[0123] The environmental boundary loss function value is obtained by summing the environmental boundary classification loss value and the environmental boundary intersection-union ratio loss value.

[0124] It should be noted that the sample parking area images are trained on multiple datasets using a multi-task architecture of "shared backbone + multi-task branches," with typical multi-task encoder-decoder inputs. First, the data passes through a data processing model. Through the above processing, sample parking area images containing results of drivable area boundaries, fixed obstacle boundaries, temporary obstacle boundaries, and real parking boundaries are obtained. All images are uniformly sized (H, W, C) = (512, 640, 3) to align with the input resolution. Common data augmentation techniques such as rotation and translation are applied to the network. The input images are then processed through a shared backbone network to extract image features, which are then fused by a feature fusion module to obtain the final image features. The obtained image features are processed through a multi-task branch model (i.e., the initial parking boundary model) to obtain predicted parking boundary results and predicted environment boundary results. The initial parking boundary model includes three segmentation branches and a regression line branch. The predicted parking boundary results and predicted environment boundary results are compared with the actual parking boundary results and actual environment boundary results obtained above to calculate the loss. The loss function of the first three segmentation branches (i.e., the environment boundary loss function value L) is used. 分割 )as follows:

[0125] L 分割 =L 分类 +L IOU ;

[0126] Among them, the environmental boundary classification loss value L 分类 It performs category classification pixel by pixel, with the environment boundary loss function value L. IOU It calculates the intersection-union ratio of two segmented regions.

[0127] Furthermore, for example, taking the environmental boundary loss function value of the drivable area boundary result as an example: L 分类 = -[y_true*log(y_pred)+(1-y_true)*log(1-y_pred)]; y_true: whether the pixel is a drivable area, 1 if yes, 0 if no; y_pred: the probability of predicting a drivable area (between 0 and 1).

[0128] In another embodiment, an environmental boundary loss function value is obtained based on the predicted environmental boundary results and the actual environmental boundary results; a parking boundary loss function value is obtained based on the predicted parking boundary results and the actual parking boundary results; and a total boundary loss function value is obtained based on the environmental boundary loss function value and the parking boundary loss function value, including:

[0129] Based on the predicted parking boundary results, the actual parking boundary results, and the preset environmental boundary classification loss algorithm, the parking boundary classification loss value is obtained;

[0130] The parking boundary distance loss value is obtained based on the distance between the predicted parking boundary result and the actual parking boundary result.

[0131] The parking boundary loss function value is obtained by summing the parking boundary classification loss value and the parking boundary distance loss value.

[0132] It should be noted that the regression branch loss function, i.e., the parking boundary loss function value L, is... 回归 for:

[0133] L 回归 =L 分类 +L 距离 ;

[0134] Here, the parking boundary classification loss value L is... 分类 This is a binary classification based on whether a pixel is a ground line (parking boundary result). The parking boundary distance loss value is the distance between the predicted parking boundary result and the actual parking boundary result. Therefore, the total boundary loss function value is as follows:

[0135] L 总 =L 分割1 +L 分割2 +L 分割3 +L 回归 ;

[0136] What I can understand is that L 分割1 L 分割2 and L 分割3 Environmental boundary loss function values ​​are derived from the drivable area boundary, fixed obstacle boundary, and temporary obstacle boundary results in the actual environmental boundary results, and from the drivable area boundary, fixed obstacle boundary, and temporary obstacle boundary results in the predicted environmental boundary results. It should be noted that L... 分割1 L 分割2 and L 分割3 The corresponding region is L. 回归 This corresponds to the boundary.

[0137] Next, obtain the current parking area image of the vehicle to be parked; input the current parking area image into the target parking boundary model to obtain the target parking boundary.

[0138] like Figure 2 As shown, based on the above method embodiments, corresponding system embodiments are provided;

[0139] Another embodiment of the present invention provides a parking boundary recognition system, comprising:

[0140] The determination module is used to determine the sample parking area image of the vehicle to be parked, and to obtain the real environment boundary result and the real parking boundary result of the sample parking area image;

[0141] The prediction module is used to obtain the predicted environment boundary results and the predicted parking boundary results based on the preset initial parking boundary model and sample parking area images;

[0142] The function acquisition module is used to obtain the environmental boundary loss function value based on the predicted environmental boundary result and the actual environmental boundary result; to obtain the parking boundary loss function value based on the predicted parking boundary result and the actual parking boundary result; and to obtain the total boundary loss function value based on the environmental boundary loss function value and the parking boundary loss function value.

[0143] The training module is used to train the initial parking boundary model based on the total boundary loss function value to obtain the target parking boundary model;

[0144] The image acquisition module is used to acquire images of the current parking area of ​​the vehicle to be parked;

[0145] The boundary recognition module is used to input the current parking area image into the target parking boundary model to obtain the target parking boundary.

[0146] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the parking boundary recognition method provided by any of the above method embodiments of the present invention.

[0147] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0148] Based on the above-described embodiment of the parking boundary recognition method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a parking boundary recognition method according to any embodiment of the present invention.

[0149] For example, in this embodiment, the computer program can be divided into one or more modules, one or more modules are stored in memory and executed by a processor to complete the present invention. One or more module elements can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.

[0150] Terminal devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. Terminal devices may include, but are not limited to, processors and memory.

[0151] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device through various interfaces and lines.

[0152] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute a parking boundary recognition method according to any of the above-described method embodiments of the present invention.

[0153] The modules / units integrated into the system / terminal device, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0154] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A parking boundary recognition method, characterized in that, include: Determine the sample parking area image of the vehicle to be parked, and obtain the real environment boundary result and the real parking boundary result of the sample parking area image; Based on the preset initial parking boundary model and the sample parking area image, the predicted environment boundary result and the predicted parking boundary result are obtained. Based on the predicted environmental boundary results and the actual environmental boundary results, the environmental boundary loss function value is obtained; based on the predicted parking boundary results and the actual parking boundary results, the parking boundary loss function value is obtained; based on the environmental boundary loss function value and the parking boundary loss function value, the total boundary loss function value is obtained. The initial parking boundary model is trained based on the total boundary loss function value to obtain the target parking boundary model; Get an image of the current parking area for the vehicle to be parked; The current parking area image is input into the target parking boundary model to obtain the target parking boundary.

2. The parking boundary recognition method according to claim 1, characterized in that, The process of determining a sample parking area image of the vehicle to be parked, and obtaining the true environmental boundary result and the true parking boundary result of the sample parking area image, includes: Determine the location information of the vehicle to be parked and the sample parking area image; Based on the preset boundary labeling model, the sample parking area image, and the location information, the real environment boundary result and the real parking boundary result are obtained.

3. The parking boundary recognition method according to claim 2, characterized in that, The real-world boundary results include drivable area boundary results. The process of obtaining the real-world boundary results and the real-world parking boundary results based on a preset boundary labeling model, the sample parking area image, and the location information includes: Based on the location information, a drivable three-dimensional area is determined; The outer contour is sampled in the drivable three-dimensional region to obtain the initial drivable three-dimensional annotation points; Acquire the extrinsic and intrinsic parameters of the imaging device for the sample parking area image; Based on the external parameter data of the shooting device and the initial 3D marking points of the drivable area, the 3D marking points of the target drivable area are obtained. Based on the intrinsic parameter data of the shooting device and the three-dimensional annotation points of the target drivable area, the two-dimensional annotation points of the drivable area are obtained; Connect the two-dimensional marker points of the drivable area to obtain the boundary result of the drivable area.

4. The parking boundary recognition method according to claim 2, characterized in that, The real environment boundary results include temporary environmental object boundary results. The process of obtaining the real environment boundary results and the real parking boundary results based on the preset boundary labeling model, the sample parking area image, and the location information includes: Based on the sample parking area image and the preset target temporary environmental object detection model, a two-dimensional bounding box for the temporary environmental object is obtained; Based on the two-dimensional bounding box of the temporary environment object and the preset segmentation model of the target temporary environment object, the boundary result of the temporary environment object is obtained.

5. The parking boundary recognition method according to claim 2, characterized in that, The process of obtaining the real environment boundary result and the real parking boundary result based on the preset boundary annotation model, the sample parking area image, and the location information includes: Obtain point cloud data of the parking area from the sample images; Based on the point cloud data of the parking area, a three-dimensional bounding box of the parking area is obtained; Based on the preset rotation and translation algorithm and the three-dimensional bounding box of the parking area, the corner points of the three-dimensional bounding box of the parking area are obtained; Filter out the bottom bounding box corner points from the corner points of the three-dimensional bounding box of the parking area; Based on the location information and the distance between the corner points of the bottom bounding box, the key parking boundary is selected from the three-dimensional bounding box of the parking area; The corner points of the bottom bounding box corresponding to the parking key boundary are transformed to obtain the actual parking boundary result.

6. The parking boundary recognition method according to claim 1, characterized in that, The environmental boundary loss function value is obtained based on the predicted environmental boundary result and the actual environmental boundary result; the parking boundary loss function value is obtained based on the predicted parking boundary result and the actual parking boundary result. Based on the environmental boundary loss function value and the parking boundary loss function value, the total boundary loss function value is obtained, including: Based on the predicted environmental boundary results, the actual environmental boundary results, and the preset environmental boundary classification loss algorithm, the environmental boundary classification loss value is obtained; Obtain the intersection area and union area of ​​the predicted environmental boundary results and the actual environmental boundary results; The environmental boundary intersection-union ratio loss value is obtained based on the ratio of the boundary intersection area to the boundary union area. The environmental boundary loss function value is obtained by summing the environmental boundary classification loss value and the environmental boundary intersection-union ratio loss value.

7. The parking boundary recognition method according to claim 1, characterized in that, The environmental boundary loss function value is obtained based on the predicted environmental boundary result and the actual environmental boundary result; the parking boundary loss function value is obtained based on the predicted parking boundary result and the actual parking boundary result. Based on the environmental boundary loss function value and the parking boundary loss function value, the total boundary loss function value is obtained, including: Based on the predicted parking boundary results, the actual parking boundary results, and the preset environmental boundary classification loss algorithm, the parking boundary classification loss value is obtained; Based on the distance between the predicted parking boundary result and the actual parking boundary result, the parking boundary distance loss value is obtained; The parking boundary loss function value is obtained by summing the parking boundary classification loss value and the parking boundary distance loss value.

8. A parking boundary recognition system, characterized in that, include: The determination module is used to determine the sample parking area image of the vehicle to be parked, and to obtain the real environment boundary result and the real parking boundary result of the sample parking area image; The prediction module is used to obtain the predicted environment boundary result and the predicted parking boundary result based on the preset initial parking boundary model and the sample parking area image. The function acquisition module is used to obtain the environmental boundary loss function value based on the predicted environmental boundary result and the actual environmental boundary result; to obtain the parking boundary loss function value based on the predicted parking boundary result and the actual parking boundary result; and to obtain the total boundary loss function value based on the environmental boundary loss function value and the parking boundary loss function value. The training module is used to train the initial parking boundary model based on the total boundary loss function value to obtain the target parking boundary model; The image acquisition module is used to acquire images of the current parking area of ​​the vehicle to be parked; The boundary recognition module is used to input the current parking area image into the target parking boundary model to obtain the target parking boundary.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a parking boundary recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a parking boundary recognition method as described in any one of claims 1-7.

Images