A visual positioning method, device and auxiliary driving system for a three-dimensional berth

Abstract

The present application discloses a visual positioning method for a three-dimensional berth, characterized in that the method comprises: obtaining a current top view image on the side of a movable body, detecting the entrance line of the three-dimensional berth based on the current top view image, and obtaining the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when the three-dimensional berth is on the ground, detecting the entry point based on the current top view image, obtaining the entry point image position information and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point for estimating the position of the three-dimensional berth, and determining the image position information and orientation of the three-dimensional berth based on the detected entrance line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located. The present application eliminates the positioning accuracy deviation of the three-dimensional berth caused by the height information, and improves the positioning accuracy in the application scenario of the three-dimensional berth.

Description

Technical Field

[0001] The invention relates to the field of machine vision, and in particular to a visual positioning method for a three-dimensional berth. Background Art

[0002] With the improvement of social productivity, ground resources are becoming increasingly scarce, and the development towards space has become a trend. In order to alleviate the limited ground resources, three-dimensional berths have been widely used.

[0003] The existing visual positioning method of stereo berths usually detects the stereo berths based on the detection algorithm of the ground berths on the corrected fisheye image. Due to the height difference between the stereo berths and the ground berths, this detection method cannot guarantee the accuracy when converting the detected berth position information from the image coordinate system to the world coordinate system. Summary of the invention

[0004] The invention provides a visual positioning method for a three-dimensional berth to improve the accuracy of positioning the three-dimensional berth on an image.

[0005] The present invention provides a visual positioning method for a three-dimensional berth, the method comprising: on the side of a movable body,

[0006] Get the current top view image.

[0007] Based on the current top view image, the entrance line of the three-dimensional berth is detected to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground.

[0008] Based on the current top view image, the entry point is detected to obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point used to estimate the position of the three-dimensional berth.

[0009] Based on the detected entry line image position information, the detected entry point image position information and the detected side line angle information of the berth where the entry point is located, the image position information and orientation of the three-dimensional berth are determined.

[0010] Preferably, the method further comprises:

[0011] Get the front view image.

[0012] Based on the front view image, it is identified whether the current scene is a three-dimensional parking scene. If the current scene is a three-dimensional parking scene, the step of acquiring the current top view image is performed.

[0013] Preferably, the acquiring the current top view image includes:

[0014] Correcting the posture of the movable body itself so that the moving direction of the movable body remains as parallel as possible to the entrance line;

[0015] When the moving direction of the movable body is kept parallel to the entrance line, a front view image, a rear view image, a left view image, and a right view image are acquired respectively.

[0016] A top view image is generated by acquiring the front view image, rear view image, left view image, and right view image.

[0017] Preferably, the detecting the entrance line of the three-dimensional berth based on the current top view image includes:

[0018] Using the trained first neural network, the endpoints of the current top view image are identified.

[0019] Perform threshold screening and non-maximum suppression operations on the identified endpoints to find feature points in the current top view image that may be the endpoints of the entry line.

[0020] Determine the feature points located on the same side of the movable body and close to the edge of the image in the moving direction as the endpoints of the entry line,

[0021] Determine a line segment defined by an endpoint close to a first edge of the image and an endpoint close to a second edge of the image as an entry line segment;

[0022] in,

[0023] The first neural network is trained as follows:

[0024] The sample top view with RGB channels is downsampled to obtain a first heat map of a single channel. In the true value of the first heat map, the closer to the endpoint of the entry line, the larger the true value.

[0025] The first heat map is input into the first neural network to train the first neural network.

[0026] Preferably, the detecting the entry point based on the current top view image comprises:

[0027] Using the trained second neural network, the entry point of the current top view image is identified to obtain image position information of the entry point, side line angle information of the berth where the entry point is located, and the confidence level of the side line angle information;

[0028] in,

[0029] The second neural network is trained as follows:

[0030] The sample top view with RGB channels is downsampled to obtain a three-channel second heat map, where the first channel represents the image position information of the entry point. The closer to the entry point, the larger the true value. The second channel represents the sine value of the side line angle of the berth where the entry point is located. The third channel represents the cosine value of the side line angle of the berth where the entry point is located.

[0031] The second heat map is input into the second neural network to train the second neural network.

[0032] Preferably, the determining of the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the detected side line angle information of the berth where the entry point is located comprises:

[0033] Perform point-line matching on the entry point and the entry line, and use the image position information of the entry point matched by the entry line to correct the entry line to obtain the corrected entry line.

[0034] For the entry points matched by the corrected entry line, determine the matching point pairs of two entry points located on the same berth.

[0035] The berth is identified by using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information.

[0036] Preferably, the performing point-line matching on the entry point and the entry line, and correcting the entry line using the image position information of the entry point matched by the entry line to obtain the corrected entry line, comprises:

[0037] For any entry point, determine whether the distance from the entry point to any entry line is within a set first distance threshold. If so, determine that the entry point matches the entry line. Traverse all entry points until each entry point is matched. Delete the unmatched entry points so that each retained entry point matches only one entry line.

[0038] Traversing all the entry lines, filtering out the entry lines whose slope differences between the entry lines are less than a set first difference threshold, the entry lines whose distances between the entry lines are less than a set second distance threshold, and the entry lines whose number of matched entry points is less than a set number threshold, and retaining one entry line;

[0039] For the retained entry line, statistics are collected on the entry point information matched by the entry line, and the first confidence of the entry line is calculated based on the statistics of the entry point information; a straight line is fitted based on all the entry points matched by the entry line, and the second confidence of the fitted straight line is calculated; the entry line is corrected based on the first confidence, the second confidence, the matched entry point information, and the degree of straight line fitting to obtain a corrected entry line.

[0040] Preferably, the step of determining a matching point pair of two entry points located on the same berth for the entry points matched by the correction entry line comprises:

[0041] According to the length of the corrected entry line and the berth setting width, the number of berths and the number of entry points are determined.

[0042] For each entry point matched by the corrected entry line, if the distance between the entry point and another entry point is within a set third distance threshold, and the difference between the side line angle of the berth where the entry point is located and the side line angle of the berth where the other entry point is located is within a set second difference threshold, then the entry point and the other entry point are determined to be a matching point pair, and the actual width of the berth is determined based on the matching point pair;

[0043] For an isolated entry point that fails to be matched successfully, an entry point adjacent to the isolated entry point among the entry points matched by the correction entry line is determined according to the actual width of the berth or the set width of the berth, as a matching point pair matched by the isolated entry point;

[0044] The number of matching point pairs is the number of entry points minus 1.

[0045] Preferably, the berth identification is performed by using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information, including:

[0046] According to the image position information of each entry point in the matching point pair, the side line angle information of the berth where it is located, and the confidence of the side line angle information, the image position of the matching point pair is corrected to the correction entry line matched by the matching point pair, and the side line angle information of the berth where each entry point in the matching point pair is located is corrected to be the same, so as to obtain a correction matching point pair, which includes two correction entry points.

[0047] Determine the berth orientation based on the angle information of the berth’s side line based on the correction matching point.

[0048] Determine the image position information of the opposite side endpoint of the correction entrance line where the correction matching point pair is located according to the side line angle information of the berth where the correction matching point pair is located, the image position information of the correction matching point pair, and the set parking space depth;

[0049] The berth mark is output based on the berth orientation, the image position information of the corrected matching point pair, and the image position information of the opposite edge endpoints.

[0050] Preferably, the three-dimensional berth is a three-dimensional parking space, and the movable body is a vehicle.

[0051] The present application also provides a visual positioning device for a three-dimensional berth, the device comprising:

[0052] The image acquisition module is used to acquire the current top view image.

[0053] The first detection module is used to detect the entrance line of the three-dimensional berth based on the current top view image to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground.

[0054] The second detection module is used to detect the entry point based on the current top view image, and obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point used to estimate the position of the three-dimensional berth.

[0055] The positioning module is used to determine the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located.

[0056] The present application also provides an assisted driving system, which includes the visual positioning device for the three-dimensional berth.

[0057] The present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the visual positioning methods for a three-dimensional berth are implemented.

[0058] The visual positioning method of the three-dimensional berth provided in the present application eliminates the positioning accuracy deviation of the three-dimensional berth caused by the height information of the three-dimensional berth by detecting the boundary line between the three-dimensional berth and the ground when the three-dimensional berth is on the ground and detecting the entry point, and obtains the image position information and orientation of the three-dimensional berth by using the image position information of the entry line and the entry point and the angle information of the side line. Since no image correction is required during the positioning process, the positioning error caused by image distortion is avoided, thereby improving the positioning accuracy in the application scenario of the three-dimensional berth. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic flow chart of a three-dimensional parking space visual positioning method according to an embodiment of the present application.

[0060] Figure 2 A schematic flow chart of a visual positioning method for a three-dimensional parking space according to an embodiment of the present application.

[0061] Figure 3 A schematic diagram of vehicle posture correction.

[0062] Figure 4 A schematic diagram of entry line detection.

[0063] Figure 5It is a schematic diagram of the position of the entrance point detected based on the current top view and the angle information of the side line of the parking space where the entrance point is located.

[0064] Figure 6 The present invention is a schematic diagram of the front view image, rear view image, left view image, right view image and the top view image generated by the panoramic surround view system during driving.

[0065] Figure 7 A schematic diagram of post-processing the entry points and entry lines detected by the neural network.

[0066] Figure 8 A schematic diagram of a visual positioning device for a three-dimensional berth according to an embodiment of the present application.

[0067] Fig. 9 A schematic diagram of an assisted driving system. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0069] The visual positioning method for a three-dimensional parking space provided in the embodiment of the present application locates the three-dimensional parking space by detecting the entrance line where the three-dimensional parking space intersects with the ground when the three-dimensional parking space is on the ground and the entrance point on the entrance line, thereby eliminating the positioning accuracy deviation caused by the height information of the three-dimensional parking space.

[0070] See also Figure 1 As shown, Figure 1 This is a flow chart of a method for visual positioning of a three-dimensional parking space according to an embodiment of the present application. The method includes:

[0071] Step 101, obtaining the current top view image,

[0072] Step 102: Based on the current top view image, the entrance line of the three-dimensional berth is detected to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground.

[0073] Step 103: Based on the current top view image, the entry point is detected to obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point used to estimate the position of the three-dimensional berth.

[0074] Step 104 : determining the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the detected side line angle information of the berth where the entry point is located.

[0075] In the above steps, steps 102 and 103 have no strict order and can be performed in parallel.

[0076] To facilitate the understanding of the present application, the visual positioning of a three-dimensional parking space in the following assisted driving application is used as an example for explanation. It should be understood that the present application is not limited to this, and the visual positioning of any three-dimensional parking space can be applied, for example, the visual positioning of a three-dimensional parking space in a mobile robot application can also be applied.

[0077] During driving, detecting available parking spaces and providing accurate location information for the subsequent parking phase is an important step in achieving autonomous driving. Unlike ground parking spaces, three-dimensional parking spaces are generally located at a different height from the driving area, which results in inaccurate locations of three-dimensional parking spaces on the image. The ground parking space detection method is not applicable to three-dimensional parking spaces.

[0078] To facilitate understanding, the basic concepts involved in this embodiment are described below.

[0079] Entrance line: The boundary line between the multi-story parking space and the ground when the multi-story parking space is located on the ground, that is, the boundary line between the multi-story parking space and the driving area.

[0080] Entry point: a feature point located on the entry line, used to estimate the position of the three-dimensional parking space, for example, the entrance corner point.

[0081] Self-vehicle: the vehicle itself.

[0082] See also Figure 2 As shown, Figure 2 The present invention is a flowchart of a visual positioning method for a three-dimensional parking space according to an embodiment of the present invention. The method includes: on the vehicle side, for example, on the auxiliary driving system side,

[0083] Step 201: Recognize a three-dimensional parking space scene based on the acquired front view image.

[0084] As an example, the vehicle is equipped with an Around View Monitor (AVM), which is a monitoring system that sets four wide-angle cameras on the front, rear, left and right of the vehicle body, and simultaneously collects four-way images, namely, front view image, rear view image, left view image and right view image. After the camera is calibrated, it can generate a 360-degree panoramic bird's-eye view around the vehicle body.

[0085] Since three-dimensional parking spaces and ground parking spaces use two different positioning methods, it is necessary to identify the scene where the vehicle is located to determine whether the vehicle has entered the three-dimensional parking space scene. One way to identify it can be:

[0086] The forward-view image acquired during driving is input into a binary classification detection network to identify the three-dimensional parking space scene and the ground parking space scene, and determine whether the vehicle has entered the three-dimensional parking space scene based on the classification result; wherein the forward-view image can be a fisheye image.

[0087] Step 202, correcting the posture of the vehicle so that the driving direction of the vehicle is as parallel to the entrance line as possible, and obtaining the current top view image when the vehicle and the entrance line remain parallel;

[0088] In view of the distortion of the height information of the three-dimensional parking spaces in the top-down view generated by the panoramic surround view system, which makes it more difficult to detect the entrance lines and entrance points of the three-dimensional parking spaces, the complexity of the sample space can be reduced and the detection difficulty can be lowered by adjusting the posture of the vehicle.

[0089] See also Figure 3 As shown, after entering the three-dimensional parking space scene, the vehicle's driving trajectory is planned according to the obstacle detection information and the ground segmentation information, and the vehicle's posture is corrected during driving so that the vehicle's driving direction is as parallel as possible to the three-dimensional parking space entrance line.

[0090] When the vehicle is parallel to the entrance line, a front view image, a rear view image, a left view image, and a right view image are acquired respectively, and a top view image is generated through the acquired front view image, rear view image, left view image, and right view image.

[0091] Step 203: Based on the acquired top view, visual inspection is performed on the parking space entrance line.

[0092] After completing the posture correction of the vehicle, the top view is used as input to detect the entrance line and entrance point of the three-dimensional parking space. In view of the fact that the three-dimensional parking space generally has height information, which will cause distortion in the top view, the entrance line and entrance point detected in this application are located at the junction of the three-dimensional parking space and the driving area, thus avoiding the deviation caused by the height information. In general, the driving area is flat and is in the same plane as the vehicle. In this way, the detected parking space entrance line can be based on the pixel distance of the entrance line detected in the top view, according to the conversion ratio between the pixel distance and the actual distance, and the world coordinates of the entrance line can be accurately calculated.

[0093] As an example, the first convolutional neural network is trained to detect the parking space entrance line. During the training process, a 320*320*3 sample top view is used as input. The sample top view has three RGB channels. The sample top view is downsampled to obtain a first heat map of 80*80*1, such as Figure 4As shown in the first heat map on the left, the true value is a floating point value between 0 and 1. The closer to the endpoint of the parking space entrance line, the larger the true value. The first heat map is a single channel. After the first convolutional neural network is trained, the trained first convolutional neural network is used to detect the endpoint of the entrance line of the currently input top view. For the endpoint detection result output by the first convolutional neural network, the feature points in the top view that may be the endpoints of the parking space entrance line are found through threshold screening and non-maximum suppression operation (NMS operation). Since the parking space entrance line is detected during the search process, the endpoint of the parking space entrance line is generally on the same side of the vehicle and is close to the edge of the image in the direction of movement of the vehicle, as shown in FIG. Figure 4 As shown in the right figure, the feature points close to the edge of the image in the v direction in the image coordinate system are Figure 4 In the right figure, the first convolutional neural network trained based on the first heat map detects the feature points of the two end points of the entrance line located on the left and right sides of the vehicle respectively. Therefore, the feature points located on the same side of the vehicle and close to the edge of the image in the moving direction are determined as the endpoints of the entrance line, and the line segment determined by the endpoint close to the first edge of the image and the endpoint close to the second edge of the image is determined as an entrance line segment. In this way, multiple entrance lines can be determined by these feature points.

[0094] Step 204, based on the acquired top view, detecting the parking space entrance point and the parking space orientation;

[0095] By detecting the parking space entrance line, we can only get the location of the single-row parking space and the approximate number of parking spaces. The role of the parking space entrance point is to infer the specific location of the parking space.

[0096] As an example, similar to the detection method of the parking space entrance line, the parking space entrance point is predicted by training the second convolutional neural network. During the training process, a 320*320*3 sample top view is used as input. The sample top view has three RGB channels. The sample top view is downsampled to obtain a second heat map of 80*80*3 as the true value. In the three channels of the second heat map, the first channel (first channel) represents the image position information of the parking space entrance point. The closer to the parking space entrance point, the larger the true value. In the last two channels, one channel (second channel) is used to predict the sine value of the angle of the side line of the parking space where the entrance point is located, and the other channel (third channel) is used to predict the cosine value of the angle of the side line of the parking space where the entrance point is located, so as to uniquely determine the angle information of the side line of the parking space through the sine value and the cosine value. The angle information is also used as the predicted direction of the entrance point and can be used to characterize the orientation of the parking space. Since there are no parking space entrance points in many places in the second heat map where the first channel is located, the true values ​​of these places are uniformly set to (-1, -1) and filtered during the training process. Through the trained second convolutional neural network, the coordinates of the parking space entrance point and the angle information of the side line of the parking space where the entrance point is located can be obtained. The angle information is used to characterize the angle between the side line of the parking space and the u direction or v direction in the image coordinate system, and the confidence of the side line angle information can also be obtained. Among them, the second convolutional neural network and the first convolutional neural network have the same analysis method for the heat map. The difference is that after obtaining the coordinates of the parking space entrance point, the angle information of the side line of the parking space where the entrance point is located is obtained according to the corresponding relationship between the predicted direction of the parking space entrance point and the parking space entrance point. The number of detected entry points is usually multiple.

[0097] See also Figure 5 As shown, Figure 5 It is a schematic diagram of the position of the entry point detected based on the current top view and the angle information of the side line of the parking space where the entry point is located. In the figure, the side line angle is the angle between the side line and the u direction of the image coordinate system.

[0098] See also Figure 6 As shown, Figure 6 The front view, rear view, left view, right view and top view images collected by the panoramic view system during driving, as well as the top view generated by these images, illustrate the entry line detection and entry point detection in the top view.

[0099] The above steps 203 and 204 have no particular order and can be performed in parallel.

[0100] Step 205, performing point-line matching on the entry point and the entry line, and using the image position information of the entry point matched by the entry line, correcting the entry line to obtain a corrected entry line.

[0101] As an example:

[0102] Step 2051, perform point-line matching on the entry lines and entry points detected by the neural network, delete the entry points that do not match the point lines, and filter the entry points.

[0103] After detecting the entrance line and entrance point information of the multi-story parking space, determine the matching relationship between the entry point and the entrance line based on the distance from the detected entry point to the entrance line and the set first distance threshold, that is, judge whether the distance from the detected entry point to the entrance line is within the first distance threshold, and if so, determine that the entry point matches the entrance line; traverse all entry points until each entry point has been matched with a point and line, ensure that each entry point only matches one entry line, and delete the entry points that have not been successfully matched.

[0104] Step 2052, filter the inlet line.

[0105] As an example, when traversing all the entrance lines, the entrance lines whose distances are less than a set second distance threshold are eliminated, and / or the entrance lines whose slope differences are less than a set first difference threshold are eliminated, and / or the entrance lines whose matched number of entry points is less than a set number threshold are eliminated, and finally one entrance line is retained. For the convenience of description, the retained entrance line is recorded as the first entrance line.

[0106] Step 2053, calibrate the retained entry line.

[0107] As an example, for the reserved entry line, that is, for the first entry line, entry point information matched by the first entry line is counted, and the confidence of the first entry line is calculated based on the counted entry point information, for example, the confidence of the first entry line is calculated based on the number of counted entry points and the distance between each counted entry point and the first entry line;

[0108] According to all the entry points matched by the first entry line, straight line fitting is performed to obtain a fitting straight line, and the confidence of the fitting straight line is calculated.

[0109] According to the confidence of the first entry line and the fitting straight line, the entry point information matched by the first entry line, and the fitting degree of the fitting straight line, the first entry line is corrected to obtain a corrected entry line, and the corrected entry line is used as the final entry line detection result. The fitting degree is determined according to the deviation distance between each entry point and the fitting straight line.

[0110] As an example, the correction of the first entrance line is expressed by the following mathematical formula.

[0111]

[0112] Among them, L is the corrected entrance line, that is, the corrected entrance line, Ll is the entrance line before correction, i.e. the first entrance line, L p is the fitting straight line, C l The first entry line L l The confidence level, C p is the fitting straight line L p The above formula can be understood as determining the weighting coefficient according to the confidence level, and performing weighted summation on the first entry line and the fitting straight line.

[0113] C l It can be determined as follows:

[0114]

[0115] Among them, C is the confidence of the entrance line itself, that is, the confidence of the entrance line detected in the first neural network detection result, β is the entrance correction coefficient, C i represents the confidence of the first entry point i matched by the line-point matching, σ i is the coefficient calculated based on the deviation distance from the entry point i to the entry line, which is used to characterize the degree of fitting. N represents the total number of matched entry points.

[0116] C p It can be determined as follows: the average confidence of the entry points matched by the first entry line-point line matching:

[0117]

[0118] Step 206, performing point-to-point matching on the entry points matched by the corrected entry line to determine the entry points belonging to the same three-dimensional parking space;

[0119] After completing point-line matching and entrance line correction, it is necessary to determine which two entrance points belong to the same parking space in preparation for parking space marking.

[0120] As an example, the number of parking spaces and the number of parking space entrance points are determined according to the corrected entrance line length and the set parking space width, for example, Figure 6 The number of parking spaces is 3 and the number of entry points is 4;

[0121] According to the distance between the entry points and the difference between the angles of the side lines of the parking spaces where the entry points are located, the matched entry points are determined, and the entry points and the matched entry points become matching point pairs. For example, for each entry point in the entry line, if the distance between the entry point and another entry point is equal to the set third distance threshold, and the difference between the angle of the side line of the parking space where the entry point is located and the angle of the side line of the parking space where the other entry point is located is less than the set second difference threshold, then the entry point and the other entry point are matching point pairs, and the actual width of the parking space can be determined according to the matching point pairs. The third distance threshold is determined according to the set width of the parking space.

[0122] For isolated entry points that fail to match successfully, the position of the entry point adjacent to the isolated entry point among the entry points matched by the correction entry line is determined according to the actual width of the parking space or the set width of the parking space. The isolated point and the adjacent entry point are a matching point pair.

[0123] The number of matching point pairs is the number of entry points minus 1.

[0124] Step 207, using the side line angle information of the parking space where each entrance point is located in the matching point pair and the image position information to mark the parking space.

[0125] Parking space identification is to correct the parking space entrance point according to the matching point pair and the side line angle, and calculate the end point position of the corrected entrance line opposite side. For example, the parking space is marked with a rectangular frame.

[0126] As an example,

[0127] According to the image position information of each entry point in the matching point pair, the side line angle information of the parking space where it is located, and the confidence of the side line angle information, the image position of the matching point pair is corrected to the corrected entry line matched by the matching point pair, and the side line angle information of the parking space where each entry point in the matching point pair is located is corrected to be the same, to obtain a corrected matching point pair, which includes two corrected entry points.

[0128] Determine the berth orientation based on the angle information of the berth’s side line based on the correction matching point.

[0129] According to the side line angle information of the berth where the correction matching point pair is located, the image position information of the correction matching point pair and the set parking space depth, according to the plane geometric relationship, determine the image position information of the opposite side end point of the correction entrance line where the correction matching point is located;

[0130] The parking space mark is output based on the parking space orientation, the image position information of the corrected matching point pair, and the image position information of the opposite edge endpoints.

[0131] Through the above steps 205 to 207, post-processing of the three-dimensional parking space is achieved.

[0132] To facilitate understanding of the processing of steps 205 to 207, see Figure 7 As shown, Figure 7 A schematic diagram of post-processing the entry points and entry lines detected by the neural network. Figure a shows the entry lines and entry points detected by the neural network; Figure b shows the point-line matching performed according to step 2051, wherein the hollow points indicate that the entry points that are not successfully matched are deleted; Figure c shows the entry line filtering performed according to step 2052, wherein the dotted lines indicate the filtered entry lines; Figure d shows the fitting straight line based on the entry points; Figure e shows that the first entry line is corrected, wherein the dotted line is the corrected entry line, i.e., the corrected entry line; Figure f shows the matching point pairs obtained according to step 206; Figure g shows the correction of the point pairs; and Figure h shows the marked parking spaces.

[0133] The visual positioning method of the three-dimensional parking space in this embodiment can eliminate the positioning accuracy deviation caused by the height information of the three-dimensional parking space through the detection of the entrance line and the entrance point, and improve the accuracy and robustness of the visual positioning through point-line matching and entrance line correction of the entrance line and the entry point, and through point pair matching and entry point correction between the entry points.

[0134] See also Figure 8 As shown, Figure 8 This is a schematic diagram of a visual positioning device for a three-dimensional berth according to an embodiment of the present application. The device includes:

[0135] The image acquisition module is used to acquire the current top view image.

[0136] The first detection module is used to detect the entrance line of the three-dimensional berth based on the current top view image to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground.

[0137] The second detection module is used to detect the entry point based on the current top view image, and obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point used to estimate the position of the three-dimensional berth.

[0138] The positioning module is used to determine the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located.

[0139] The device also includes:

[0140] The scene recognition module is used to obtain a forward-looking perspective image, and based on the forward-looking perspective image, identify whether the current scene is a stereoscopic parking scene, and select a stereoscopic parking positioning method when the current scene is a stereoscopic parking scene.

[0141] The posture correction module is used to correct the posture of the movable body itself so that the moving direction of the movable body remains as parallel as possible to the entrance line.

[0142] in,

[0143] The image acquisition module is configured to respectively acquire a front view perspective image, a rear view perspective image, a left view perspective image, and a right view perspective image when the moving direction of the movable body remains parallel to the entrance line, and generate a top view image through the acquired front view perspective image, rear view perspective image, left view perspective image, and right view perspective image.

[0144] The first detection module is configured to use the trained first neural network to perform endpoint recognition on the current top view image.

[0145] Perform threshold screening and non-maximum suppression operations on the identified endpoints to find feature points in the current top view image that may be the endpoints of the entry line.

[0146] Determine the feature points located on the same side of the movable body and close to the edge of the image in the moving direction as the endpoints of the entry line,

[0147] A line segment defined by an endpoint close to a first edge of the image and an endpoint close to a second edge of the image is determined as an entry line segment.

[0148] The second detection module is configured to use the trained second neural network to identify the entry point of the current top view image, obtain the image position information of the entry point, the side line angle information of the berth where the entry point is located, and the confidence level of the side line angle information.

[0149] The positioning module is configured to perform point-line matching on the entry point and the entry line, and to correct the entry line using the image position information of the entry point matched by the entry line to obtain a corrected entry line.

[0150] For the entry points matched by the corrected entry line, determine the matching point pairs of two entry points located on the same berth.

[0151] The berth is identified by using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information.

[0152] in,

[0153] The positioning module includes:

[0154] An entry line correction submodule is used to determine, for any entry point, whether the distance from the entry point to any entry line is within a set first distance threshold. If so, the entry point is determined to match the entry line, and all entry points are traversed until each entry point is matched, and the unmatched entry points are deleted, so that each retained entry point matches only one entry line;

[0155] Traversing all the entry lines, filtering out the entry lines whose slope differences between the entry lines are less than a set first difference threshold, the entry lines whose distances between the entry lines are less than a set second distance threshold, and the entry lines whose number of matched entry points is less than a set number threshold, and retaining one entry line;

[0156] For the retained entry line, statistics are collected on the entry point information matched by the entry line, and the first confidence of the entry line is calculated based on the statistics of the entry point information; a straight line is fitted based on all the entry points matched by the entry line, and the second confidence of the fitted straight line is calculated; the entry line is corrected based on the first confidence, the second confidence, the matched entry point information, and the degree of straight line fitting to obtain a corrected entry line.

[0157] The entry point pair matching submodule is used to determine the number of berths and the number of entry points according to the length of the corrected entry line and the berth setting width.

[0158] For each entry point matched by the corrected entry line, if the distance between the entry point and another entry point is within a set third distance threshold, and the difference between the side line angle of the berth where the entry point is located and the side line angle of the berth where the other entry point is located is within a set second difference threshold, then the entry point and the other entry point are determined to be a matching point pair, and the actual width of the berth is determined based on the matching point pair;

[0159] For an isolated entry point that fails to be matched successfully, an entry point adjacent to the isolated entry point among the entry points matched by the correction entry line is determined according to the actual width of the berth or the set width of the berth, as a matching point pair matched by the isolated entry point;

[0160] The parking space recognition submodule is used to correct the image position of the matching point pair to the corrected entry line matched by the matching point pair according to the image position information of each entry point in the matching point pair, the side line angle information of the berth where the entry point is located, and the confidence of the side line angle information, and correct the side line angle information of the berth where each entry point in the matching point pair is located to be the same, so as to obtain a corrected matching point pair, wherein the corrected matching point pair includes two corrected entry points.

[0161] Determine the berth orientation based on the angle information of the berth’s side line based on the correction matching point.

[0162] Determine the image position information of the opposite side endpoint of the correction entrance line where the correction matching point pair is located according to the side line angle information of the berth where the correction matching point pair is located, the image position information of the correction matching point pair, and the set parking space depth;

[0163] The berth mark is output based on the berth orientation, the image position information of the corrected matching point pair, and the image position information of the opposite edge endpoints.

[0164] The visual positioning device for a three-dimensional parking space of this embodiment can be applied to an auxiliary driving system of a vehicle.

[0165] See also Fig. 9 As shown, Fig. 9 The figure is a schematic diagram of an assisted driving system. The system includes a panoramic surround view system, a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the visual positioning method of the three-dimensional berth in the embodiment of the present application.

[0166] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual positioning method of a three-dimensional berth are implemented.

[0169] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0171] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visual positioning method for a three-dimensional berth, characterized in that: The method comprises: on the movable body side, Get the current top view image. Based on the current top view image, the entrance line of the three-dimensional berth is detected to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground. Based on the current top view image, the entry point is detected to obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point for estimating the position of the three-dimensional berth and is located at the boundary line. Determining the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located; in, The entrance line of the three-dimensional berth is detected in the following way: Using the trained first neural network, the endpoints of the current top view image are identified. Based on the identified endpoints, find the feature points in the current top view image that may be the endpoints of the entry line. Determine the feature points in the current top view image that are located on the same side of the movable body and close to the edge of the image in the moving direction as the endpoints of the entry line, Determine a line segment defined by an endpoint close to a first edge of the image and an endpoint close to a second edge of the image as an entry line segment; The first neural network is trained as follows: Inputting the first heat map into the first neural network to train the first neural network, wherein the closer the true value of the first heat map is to the endpoint of the entry line, the greater the true value; The method of determining the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located includes: Perform point-line matching on the entry point and the entry line, and use the image position information of the entry point matched by the entry line to correct the entry line to obtain the corrected entry line. For the entry points matched by the corrected entry line, determine the matching point pairs of two entry points located on the same berth. The berth is identified by using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information.

2. The visual positioning method according to claim 1, characterized in that: The method further comprises: A forward-viewing perspective image is acquired, and based on the forward-viewing perspective image, whether the current scene is a three-dimensional parking scene is identified. If the current scene is a three-dimensional parking scene, the step of acquiring the current top-view image is performed.

3. The visual positioning method according to claim 1, characterized in that: The step of obtaining the current top view image includes: Correcting the posture of the movable body itself so that the moving direction of the movable body remains as parallel as possible to the entrance line; When the moving direction of the movable body is kept parallel to the entrance line, a front view image, a rear view image, a left view image, and a right view image are acquired respectively. A top view image is generated by acquiring the front view image, rear view image, left view image, and right view image.

4. The visual positioning method according to claim 3, characterized in that: The step of finding a feature point in the current top view image that may be an endpoint of the entry line based on the identified endpoint includes: Perform threshold screening and non-maximum suppression operations on the identified endpoints to find feature points in the current top view image that may be endpoints of the entry line; The first thermal map is obtained in the following manner: The sample top view with RGB channels is downsampled to obtain the first heat map of a single channel.

5. The visual positioning method according to claim 3, characterized in that: The detecting the entry point based on the current top view image includes: The trained second neural network is used to identify the entry point of the current top view image to obtain image position information of the entry point, side line angle information of the berth where the entry point is located, and the confidence level of the side line angle information.

6. The visual positioning method according to claim 5, characterized in that: The second neural network is trained as follows: The sample top view with RGB channels is downsampled to obtain a three-channel second heat map, where the first channel represents the image position information of the entry point. The closer to the entry point, the larger the true value. The second channel represents the sine value of the side line angle of the berth where the entry point is located. The third channel represents the cosine value of the side line angle of the berth where the entry point is located. The second heat map is input into the second neural network to train the second neural network.

7. The visual positioning method according to claim 1, characterized in that: The step of performing point-line matching on the entry point and the entry line, and correcting the entry line using the image position information of the entry point matched by the entry line to obtain the corrected entry line, comprises: For any entry point, determine whether the distance from the entry point to any entry line is within a set first distance threshold. If so, determine that the entry point matches the entry line. Traverse all entry points until each entry point is matched. Delete the unmatched entry points so that each retained entry point matches only one entry line. Traversing all the entry lines, filtering out the entry lines whose slope differences between the entry lines are less than a set first difference threshold, the entry lines whose distances between the entry lines are less than a set second distance threshold, and the entry lines whose number of matched entry points is less than a set number threshold, and retaining one entry line; For the retained entry line, statistics are collected on the entry point information matched by the entry line, and the first confidence of the entry line is calculated based on the statistics of the entry point information; a straight line is fitted based on all the entry points matched by the entry line, and the second confidence of the fitted straight line is calculated; the entry line is corrected based on the first confidence, the second confidence, the matched entry point information, and the degree of straight line fitting to obtain a corrected entry line.

8. The visual positioning method according to claim 1, characterized in that: The matching point pair of two entry points located at the same berth is determined by matching the entry points of the corrected entry line, including: According to the length of the corrected entry line and the berth setting width, the number of berths and the number of entry points are determined. For each entry point matched by the corrected entry line, if the distance between the entry point and another entry point is within a set third distance threshold, and the difference between the side line angle of the berth where the entry point is located and the side line angle of the berth where the other entry point is located is within a set second difference threshold, then the entry point and the other entry point are determined to be a matching point pair, and the actual width of the berth is determined based on the matching point pair; For an isolated entry point that fails to be matched successfully, an entry point adjacent to the isolated entry point among the entry points matched by the correction entry line is determined according to the actual width of the berth or the set width of the berth, as a matching point pair matched by the isolated entry point; The number of matching point pairs is the number of entry points minus 1.

9. The visual positioning method according to claim 1, characterized in that: The method of using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information to identify the berth includes: According to the image position information of each entry point in the matching point pair, the side line angle information of the berth where it is located, and the confidence of the side line angle information, the image position of the matching point pair is corrected to the correction entry line matched by the matching point pair, and the side line angle information of the berth where each entry point in the matching point pair is located is corrected to be the same, so as to obtain a correction matching point pair, which includes two correction entry points. Determine the berth orientation based on the angle information of the berth’s side line based on the correction matching point. Determine the image position information of the opposite side endpoint of the correction entrance line where the correction matching point pair is located according to the side line angle information of the berth where the correction matching point pair is located, the image position information of the correction matching point pair, and the set parking space depth; The berth mark is output based on the berth orientation, the image position information of the corrected matching point pair, and the image position information of the opposite edge endpoints.

10. The visual positioning method according to claim 1, characterized in that: The three-dimensional berth is a three-dimensional parking space, and the movable body is a vehicle.

11. A visual positioning device for a three-dimensional berth, characterized in that: The device includes: The image acquisition module is used to acquire the current top view image. The first detection module is used to detect the entrance line of the three-dimensional berth based on the current top view image to obtain the entrance line image position information, wherein the entrance line is the boundary line between the three-dimensional berth and the ground when it is on the ground. The second detection module is used to detect the entry point based on the current top view image, and obtain the image position information of the entry point and the side line angle information of the berth where the entry point is located, wherein the entry point is a feature point for estimating the position of the three-dimensional berth, and is located at the intersection line. A positioning module, for determining the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located; in, The entrance line of the three-dimensional berth is detected in the following way: Using the trained first neural network, the endpoints of the current top view image are identified. Based on the identified endpoints, find the feature points in the current top view image that may be the endpoints of the entry line. The feature points in the current top view image that are located on the same side of the movable body and close to the edge of the image in the moving direction are determined as the endpoints of the entry line. Determine a line segment defined by an endpoint close to a first edge of the image and an endpoint close to a second edge of the image as an entry line segment; The first neural network is trained as follows: Inputting the first heat map into the first neural network to train the first neural network, wherein the closer the true value of the first heat map is to the endpoint of the entry line, the greater the true value; The method of determining the image position information and orientation of the three-dimensional berth based on the detected entry line image position information, the detected entry point image position information and the side line angle information of the berth where the detected entry point is located includes: Perform point-line matching on the entry point and the entry line, and use the image position information of the entry point matched by the entry line to correct the entry line to obtain the corrected entry line. For the entry points matched by the corrected entry line, determine the matching point pairs of two entry points located on the same berth. The berth is identified by using the side line angle information of the berth where each entry point is located in the matching point pair and the image position information.

12. A driving assistance system, characterized in that: The system comprises the visual positioning device for the three-dimensional berth as claimed in claim 11.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual positioning method of a three-dimensional berth as described in any one of claims 1 to 10 are implemented.

Images