Parking Space Positioning Method, System, Medium, and Driverless Sanitation Robot

Abstract

The present invention discloses a parking space positioning method, system, medium, and driverless sanitation robot. In the present invention, the midpoint of the parking space area is set at the midpoint of the detection frame, and the parking space border line is set on the diagonal or the center line of the detection frame. Such a label generation method enables the trained model to directly detect the parking space border line and its position in the image. For environments where the corners of the parking space are unclear or there are stains, especially for such environments, the overall information of the parking space border line can be effectively extracted for recognition, the spatial relationship of the parking space corner points is retained, and there is high stability, and the calculation efficiency is also improved. Also, according to the recognition result of the model and using the spatial structure of the parking space for conditional constraints, a parking space with a complete four sides is generated, without considering the angle of the parking space and without the need to perform straight line detection, which can improve the detection accuracy, reduce the missed detection rate, and also has a great improvement in terms of generality.

Description

Technical Field

[0001] The present invention relates to the technical field of parking space recognition, and particularly to a parking space positioning method, system, medium, and driverless sanitation robot. Background Art

[0002] At present, the detection of the spatial position of parking spaces on the market mainly includes edge straight line detection based on traditional vision algorithms, fixed template matching, and key point detection:

[0003] The edge straight line detection based on traditional vision algorithms is greatly interfered by the scene. For example, using traditional vision algorithms to detect the position of parking spaces will be affected by unknown factors such as environmental illumination and ground stains. As Figure 1 shown, if the contrast of the detection border of the parking space decreases due to long-term use, it will lead to recognition failure. As Figure 2 shown, fixed template matching will result in poor generality of recognition, that is, only single types of parking spaces can be recognized. As Figure 3 and Figure 4 shown, it is difficult to obtain the spatial relationship between the detected points by using the deep learning key point detection method, so it is necessary to rely on the assistance of other algorithms, resulting in limited accuracy. Summary of the Invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art. For this purpose, the present invention provides a parking space positioning method, system, medium, and driverless sanitation robot, which can extract the overall information of the parking space border and identify it, retain the spatial relationship of the corner points of the parking space, have high stability, and can also improve the detection accuracy and calculation efficiency.

[0005] In a first aspect of the present invention, a parking space positioning method is provided, including the following steps:

[0006] Obtain a parking space area in a parking space image, and mark the parking space entrance corner point, the inner corner points of the parking space area, and the midpoint, where the inner corner points of the parking space area refer to the inner corner points of the parking space or the endpoints of the border of the parking space area close to the inner corner points of the parking space;

[0007] Generate a training label according to the marked parking space entrance corner point, the inner corner points of the parking space area, and the midpoint. Among them, set the midpoint of the parking space area at the midpoint of the detection border, and set the corresponding parking space border on the diagonal or the center line of the detection border according to the coordinate transformation of the parking space entrance corner point and the inner corner points of the parking space area;

[0008] Train a deep convolutional neural network model through the marked parking space image and the training label to obtain the recognition result output by the deep convolutional neural network model;

[0009] Generate a parking space with complete four sides according to the recognition result.

[0010] According to the embodiments of the present invention, there are at least the following technical effects:

[0011] This method first obtains the corner points of the parking space entrance, the interior corner points and the midpoint of the parking space area within the image, then sets the midpoint of the parking space area at the midpoint of the detection border, and uses the coordinate transformation of the corner points of the parking space entrance and the interior corner points of the parking space area to set the corresponding parking space side lines on the diagonal or the midline of the detection border. Then, use the generated labels and the annotated images to train the deep convolutional neural network model. Finally, according to the recognition result of the model and using the spatial structure of the parking space for conditional constraints, generate a parking space with complete four sides. This method sets the midpoint of the parking space area at the midpoint of the detection border and sets the parking space side lines on the diagonal or the midline of the detection border. Such a label generation method enables the trained model to directly detect the parking space side lines and their positions in the image. For environments where the corners of the parking space are not clear or there are stains, especially for such environments, it can effectively extract the overall information of the parking space side lines for recognition, retains the spatial relationship of the corner points of the parking space, has high stability, and also improves the calculation efficiency. This method generates a parking space with complete four sides according to the recognition result of the model and using the spatial structure of the parking space for conditional constraints, without considering the angle of the parking space and without the need to perform straight line detection, which can improve the detection accuracy, reduce the missed detection rate, and also has a great improvement in terms of generality.

[0012] According to some embodiments of the present invention, the setting of the parking space side lines on the diagonal or the midline of the detection border according to the coordinate transformation of the corner points of the parking space entrance and the interior corner points of the parking space area includes:

[0013] Calculate the angle between the corresponding parking space side line and the bottom edge of the parking space image according to the corner points of the parking space entrance and the interior corner points of the parking space area;

[0014] When the parking space side line is perpendicular or parallel to the bottom edge of the parking space image, set the parking space side line at the midpoint of the opposite side of the detection border through coordinate transformation; when the parking space side line is neither perpendicular nor parallel to the bottom edge of the parking space image, set the parking space side line on the diagonal of the detection border through coordinate transformation.

[0015] According to some embodiments of the present invention, the deep convolutional neural network model is the YoloV5 network model.

[0016] According to some embodiments of the present invention, the training of the deep convolutional neural network model by the annotated parking space image and the training labels includes:

[0017] Obtain all the recognized parking space sidelines;

[0018] Calculate the Euclidean distance between the current parking space sideline and each of the remaining parking space sidelines to obtain a set D e ={d n |n = 1, 2, …, m}, where d n represents the Euclidean distance between the current parking space sideline and one of the remaining parking space sidelines, and m represents the number of parking space sidelines;

[0019] Select from the set D s ={d n |d n <d min , d n ∈D e} the element with the highest credibility as the parking space sideline corresponding to a regional position in the parking space image, where d min is a custom threshold.

[0020] According to some embodiments of the present invention, generating a parking space with a complete four sides based on the recognition result includes:

[0021] Obtain all the recognized parking spaces;

[0022] Calculate the Euclidean distance between the current parking space and each of the recognized parking space sidelines to obtain a distance set;

[0023] Select two parking space sidelines corresponding to the two smallest Euclidean distances in the distance set;

[0024] When both of the two parking space sidelines are less than a preset value, combine the two parking space sidelines and the corresponding endpoints of the two parking space sidelines to obtain the four sidelines of the current parking space.

[0025] According to some embodiments of the present invention, during the training process of the deep convolutional neural network model, delete the flipping and rotation processing of the parking space image.

[0026] According to some embodiments of the present invention, before obtaining the parking space area located in the parking space image, it further includes:

[0027] Fuse the images captured by the four camera devices in the front, back, left, and right of the driverless sanitation robot to obtain a parking space image.

[0028] In a second aspect of the present invention, there is provided a parking space positioning system, including:

[0029] A data annotation unit for obtaining a parking space area located in a parking space image, annotating the corner points of the parking space entrance, the interior corner points and the midpoint of the parking space area, where the interior corner points of the parking space area refer to the inner corner points of the parking space or the endpoints of the side lines of the parking space area close to the inner corner points of the parking space;

[0030] A label generation unit for generating training labels according to the annotated corner points of the parking space entrance, the interior corner points and the midpoint of the parking space area. Among them, the midpoint of the parking space area is set at the midpoint of the detection frame, and the corresponding parking space side lines are set on the diagonal or the midline of the detection frame according to the coordinate transformation of the corner points of the parking space entrance and the interior corner points of the parking space area;

[0031] A model training unit for training a deep convolutional neural network model through the annotated parking space image and the training labels to obtain the recognition result output by the deep convolutional neural network model;

[0032] A parking space generation unit for generating a parking space with complete four sides according to the recognition result.

[0033] In the third aspect of the present invention, there is provided a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-mentioned parking space positioning method.

[0034] In the fourth aspect of the present invention, there is provided an unmanned sanitation robot including at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the above-mentioned parking space positioning method.

[0035] It should be noted that the beneficial effects between the second to fourth aspects of the present invention and the prior art are the same as those between the above-mentioned parking space positioning method and the prior art, and will not be elaborated here.

[0036] The additional aspects and advantages of the present invention will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0038] Figure 1 It is a flowchart of a related solution for parking space recognition based on a traditional vision-based straight line detection algorithm;

[0039] Figure 2 Template diagram of the relevant solution for a parking space matching a fixed parking space type

[0040] Figure 3 Flow chart of the relevant solution for parking space recognition that combines the key point detection technology of deep learning and the matching of fixed parking space types

[0041] Figure 4 Schematic diagram of the relevant solution for parking space recognition that combines the key point detection technology of deep learning and the traditional vision straight line detection algorithm

[0042] Figure 5 Schematic flow chart of a parking space positioning method provided by an embodiment of the present invention

[0043] Figure 6 Schematic diagram for marking the entrance corner point, inner corner point of the parking space, and midpoint of the parking space in the parking space image provided by an embodiment of the present invention

[0044] Figure 7 Schematic diagram for generating a detection border provided by an embodiment of the present invention

[0045] Figure 8 Schematic diagram of the recognition result of the deep convolutional neural network model provided by an embodiment of the present invention

[0046] Figure 9 Schematic flow chart of a parking space positioning method provided by another embodiment of the present invention

[0047] Figure 10 Schematic diagram of the structure of a parking space recognition system provided by an embodiment of the present invention Detailed implementation manners

[0048] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0049] Referring to Figures 5 to 7 , an embodiment of the present invention provides a parking space positioning method. This method can be used for detecting the spatial position of a parking space during automatic parking for driverless or assisted driving. This method includes the following steps:

[0050] Step S110: Obtain the parking space area in the parking space image, and mark the entrance corner point of the parking space, the inner corner point of the parking space area, and the midpoint. The inner corner point of the parking space area refers to the inner corner point of the parking space or the end point of the side line of the parking space area close to the inner corner point of the parking space.

[0051] In this embodiment, the parking space image (i.e., picture) is taken by an autonomous sanitation robot. For example, the annular perspective image is constructed by taking images of the parking lot or roadside parking spaces through four camera devices (such as cameras) located at the front, rear, left, and right of the autonomous sanitation robot and then fusing them. Multiple parking space images can be obtained by the autonomous sanitation robot to form a dataset for training the model. It should be noted that there is a parking space area in the parking space image. However, since part of the parking space is outside the image, the parking space area described in step S110 is not necessarily complete. It should also be noted that the parking space boundary line is a well-known feature to those skilled in the art. Due to the situation that part of the parking space is outside the image, for the sake of easy understanding, the boundary line of the parking space area is used here in step S110 to describe the parking space boundary line. After obtaining the parking space image, it is necessary to annotate the image, mainly annotating the four corner points and the midpoint of the parking space. The annotation process is as follows:

[0052] First, annotate two types of corner points, that is, annotate the parking space entrance corner point and the interior corner point of the parking space area in step S110;

[0053] (1) When the parking space in the image is complete (i.e., the parking space area in the image is complete), annotate the parking space entrance corner point and the inner corner point of the parking space. It should be noted that the midpoint of the parking space, the parking space entrance corner point, and the inner corner point of the parking space are well-known features to those skilled in the art.

[0054] (2) When the parking space in the image is incomplete (i.e., part of the parking space is outside the image and the parking space area in the image is not complete), annotate the parking space entrance corner point (regardless of whether the parking space area in the image is complete or not, the parking space entrance corner point can be annotated). Since the inner corner point of the parking space is missing in the parking space area in the image, the end point of the boundary line of the parking space area close to the inner corner point of the parking space is used as the annotated corner point (i.e., the interior corner point of the parking space area described in step S110).

[0055] Second, annotate the midpoint of the parking space area. The parking space includes an empty parking space (i.e., an idle parking space) and an occupied parking space (a parking space with a vehicle parked);

[0056] (1) When the parking space in the image is complete, directly annotate the midpoint of the parking space. In this case, the midpoint of the parking space area described in step S110 is the same as the midpoint of the parking space.

[0057] (2) When the parking space in the image is incomplete, since the parking space area in the image is incomplete, the midpoint of the current parking space area is used as the annotation point.

[0058] Step S120: Generate training labels based on the marked corner points of the parking space entrance, the interior corner points and the midpoint of the parking space area. Among them, set the midpoint of the parking space area at the midpoint of the detection border, and set the corresponding parking space border on the diagonal or the center line of the detection border according to the coordinate transformation of the corner points of the parking space entrance and the interior corner points of the parking space area.

[0059] In some embodiments, setting the parking space border on the diagonal or the center line of the detection border specifically includes:

[0060] Step S1201: Calculate the angle between the corresponding parking space border and the bottom edge of the parking space image according to the corner points of the parking space entrance and the interior corner points of the parking space area.

[0061] Step S1202: When the parking space border is perpendicular or parallel to the bottom edge of the parking space image, set the parking space border at the midpoint of the opposite side of the detection border through coordinate transformation; when the parking space border is neither perpendicular nor parallel to the bottom edge of the parking space image, set the parking space border on the diagonal of the detection border through coordinate transformation.

[0062] For example, calculate the included angle between the parking space border and the bottom edge of the parking space image according to the coordinates of the corner points of the parking space entrance and the interior corner points of the parking space area, and then use the threshold method to divide different included angle ranges. For example, let the included angle between the parking space border and the bottom edge of the parking space image be α. When α < -85 or α > -85, the parking space border is perpendicular to the bottom edge of the parking space image; when -5 < α < 5, the parking space border is parallel to the bottom edge of the parking space image; when 5 < α < 85 or -85 < α < -5, the parking space border is neither perpendicular nor parallel to the bottom edge of the parking space image.

[0063] Refer to Figure 7 , Step S120 generates training labels through the transformation of the coordinate data of the two types of marked corner points, aiming to make the corresponding parking space border be on the diagonal or the center line of the detection border (i.e., the training label).

[0064] Step S130: Train the deep convolutional neural network model through the marked parking space image and the training label to obtain the recognition result output by the deep convolutional neural network model.

[0065] In the above step S120, setting the midpoint of the parking space area at the midpoint of the detection border and setting the parking space border on the diagonal or the center line of the detection border, such a label generation method enables the trained model to directly detect the parking space border and its position in the image. For the environment where the corners of the parking space are not clear or there are stains, especially for the environment where the corners of the parking space are not clear or there are stains, it can effectively extract the overall information of the parking space border for recognition, retain the spatial relationship of the corner points of the parking space, have high stability, and also improve the calculation efficiency.

[0066] In some embodiments, the deep convolutional neural network model used in this embodiment can be R-CNN, SSD, RetinaNet, and YoloV5 network models. Preferably, the deep convolutional neural network model used in this embodiment is the YoloV5 network model. According to the general training process of the YoloV5 network model (i.e., using the method of stochastic gradient descent and updating the network parameter weights by backpropagation for training), the Batch size (the number of samples selected for one training) is set to 300, and the epochs (single training iterations for all batches in forward and backward propagation) are set to 50. Using the YoloV5 network model to detect lines through a rectangular box, the detection of lines by the rectangular box depends heavily on the orientation of the lines in the image, and some operations of data augmentation will change the orientation of the lines in the training data, such as flipping and rotation. Therefore, in the training process of the YoloV5 network model in this embodiment, the two data augmentation methods of image flipping and rotation are directly cancelled, which can improve the detection accuracy and calculation efficiency and enhance the recognition accuracy.

[0067] According to the related solution, there will be a misjudgment rate in the inference process of the YoloV5 network model. For example, for the detection of the parking space boundary line, the same boundary line will have a category change due to the rotation of the vehicle. If the accuracy of the model is not high enough, it may lead to the situation of judging a boundary line as multiple types under the same confidence threshold, resulting in misjudgment. In some embodiments of the present invention, in order to improve the recognition accuracy of the model, the adjacent heterogeneous non-maximum suppression method is applied to the training process of the YoloV5 network model. Taking the model's recognition of the parking space boundary line as an example, the specific process is as follows:

[0068] According to the spatial structure characteristics of the parking space, it can be known that there is exactly one parking space boundary line within a certain area range. First, obtain all the parking space boundary lines detected in the parking space image.

[0069] Perform iterative calculations on each parking space boundary line. Taking the current parking space boundary line as an example, calculate the Euclidean distance between the current parking space boundary line and each of the remaining parking space boundary lines to obtain the set D e ={d n |n = 1, 2, …, m}, where d n represents the Euclidean distance between the current parking space boundary line and one of the remaining parking space boundary lines, and m represents the number of parking space boundary lines.

[0070] From the set D s ={d n |d n <d min , d n ∈D e} Select the element with the highest credibility as the parking space boundary line corresponding to the regional position in the parking space image, where d min is a custom threshold.

[0071] By applying the adjacent heterogeneous non-maximum suppression method to the training process of the YoloV5 network model, the accuracy of model recognition can be greatly improved.

[0072] Step S140: Generate a parking space with complete four sides according to the recognition result.

[0073] In this step, according to the spatial structure of the parking space, perform conditional constraints, select two corresponding parking space boundary lines from all the recognized parking space boundary lines, and generate a parking space with complete four sides based on the two parking space boundary lines and the corresponding endpoints of the two parking space boundary lines. The specific steps are as follows:

[0074] Step S1401: Obtain all the recognized parking spaces.

[0075] Step S1402: Calculate the Euclidean distance between the current parking space and each recognized parking space boundary line to obtain a distance set.

[0076] Step S1403: Select two parking space boundary lines corresponding to the two smallest Euclidean distances in the distance set.

[0077] Step S1404: When both parking space boundary lines are less than the preset value, combine the two parking space boundary lines and the corresponding endpoints of the two parking space boundary lines to obtain the four boundary lines of the current parking space.

[0078] According to the relevant solution, there is a phenomenon that the parking space boundary lines near the parking space that do not belong to this parking space form a wrong parking space. To avoid this situation, in steps S1401 to S1404, for any parking space detected by the model, calculate the two parking space boundary lines closest to this parking space. If both of these two parking space boundary lines are less than the preset value, then a parking space with a complete four sides (including two parking space boundary lines, one boundary line at the parking space entrance, and one bottom boundary line near the inside) can be formed by these two parking space boundary lines and their two parking space entrance corner points and two parking space inner corner points. It can solve the phenomenon that the parking space boundary lines near the parking space that do not belong to this parking space form a wrong parking space, and compared with the relevant solutions (such as Figure 3 and Figure 4 ), there is no need to perform line detection anymore, and a parking space with complete four sides can be constructed through simple calculations. On the premise of ensuring high accuracy and low missed detection rate, the requirements for computing devices are reduced.

[0079] For the convenience of understanding by those skilled in the art, refer to Figures 6 to 9, a set of best embodiments are provided here, providing a parking space positioning method. This method is mainly applied to the recognition scenarios of unmanned sanitation robots in indoor parking lots or roadside parking spaces, and uses deep learning object detection technology for line detection. The specific process is as follows:

[0080] Step S210: Use the annular perspective image constructed by fusing the four cameras in the front, back, left, and right of the unmanned sanitation robot, and construct a dataset for model training, marking the coordinates of the four corner points of the parking space (two entrance corner points of the parking space and the inner corner point of the parking space) and the midpoint of the parking space in the image. In the image, the data types are divided into four types, namely the entrance corner point and the inner corner point on the parking space boundary line, empty parking spaces, and occupied parking spaces. Step S210 specifically includes:

[0081] Step S2101: Mark the entrance corner point and the inner corner point of the parking space on the parking space boundary line. It should be noted that when marking the parking space boundary line, the corner points of each boundary line can be marked separately, and there is no need to give the label information constituting a complete parking space. When marking the inner corner point of the parking space, if the inner corner point of the parking space is outside the image, then the point on the parking space boundary line closest to this corner point in the image (usually at the edge of the image, as Figure 6 shown) is used as the inner corner point of the parking space.

[0082] Step S2102: Mark empty parking spaces and occupied parking spaces, and the position of the marked point is at the center point of the empty parking space or the occupied parking space. When a part of the parking space is outside the image, then the midpoint of the parking space area within the image (as Figure 6 shown) is used as the midpoint of the parking space.

[0083] Step S220: Generate training labels through the coordinate transformation after marking. The purpose of this step is to make the corresponding parking space boundary line on the diagonal or the midline of the detection frame, and make the midpoint of the parking space at the midpoint of the detection frame.

[0084] Convert the points marked in step S210 into the label format based on deep learning object detection. The main idea of this part is to attach the parking space boundary line perpendicular or parallel to the bottom edge of the image to the midpoint of the opposite side of the detection frame, and attach the parking space boundary line at the remaining angles to the bottom edge of the image to the diagonal of the detection frame (as Figure 7 shown). Such marking can not only allow the trained neural network to directly detect the boundary line of the parking space and its position in the figure, but also for the environment where the corners of the parking space are not clear or there are stains, the neural network can effectively extract the overall information of the parking space boundary line for recognition, retaining the spatial relationship of the corner points of the parking space, with high stability. Specifically, the processing steps are as follows:

[0085] Step S2201: Let the set of parking space boundary lines in an image be \(E = \{e_{n}|n = 1,2,\cdots,m\}\), where \(e_{n}=\{p_{n1},p_{n2}\}\), \(p_{n1}\) represents the entrance corner point of the parking space, and \(p_{n2}\) represents the inner corner point of the parking space. n | n where o , i} o , i represents the inner corner point of the parking space.

[0086] Step S2202: Let the horizontal and vertical coordinates of \(p_{n1}\) be \(x_{n1},y_{n1}\), and the horizontal and vertical coordinates of \(p_{n2}\) be \(x_{n2},y_{n2}\). Derive the angle between the parking space boundary line \(e_{n}\) and the bottom edge of the image through the following formula: o where o , o , i where i , i , n and the bottom edge of the image:

[0087]

[0088] where \(\alpha\) is the degree of the included angle.

[0089] Step S2203: Divide the parking space boundary lines into four categories according to the angle between the parking space boundary line and the bottom edge of the image. Let the four categories be \(c1,c2,c3,c4\). \(c1:\alpha < - 85\) or \(\alpha> - 85\) (the parking space boundary line is perpendicular to the bottom edge of the image); \(c2:-5 <\alpha < 5\) (the parking space boundary line is parallel to the bottom edge of the image); \(c3:5 <\alpha < 85\); \(c4:-85 <\alpha < - 5\).

[0090] Step S2204: Calculate the label data. The empty parking space and the occupied parking space are regarded as two different categories. Together with the four categories of parking space boundary lines, there are a total of six categories. In this embodiment, YoloV5 is used as the deep neural network. According to the label format of YoloV5, the label data can be calculated using the following formula:

[0091]

[0092] L slot =\{x s ,y s ,w s ,h s \}

[0093] where line is the label of the parking space boundary line, slot is the label of the parking space, s , s are the horizontal and vertical coordinates of the empty parking space or the occupied parking space respectively, s , s are the width and length of the parking space, s ,s The calculation method is as follows:

[0094] w s = min(x s , s max , w I - x s )

[0095] h s = min(y s , s max , h I - x s )

[0096] Wherein, s max is the maximum side length of the custom parking space detection border, and w I , h I are the width and length of the image.

[0097] It should be noted that when the detection border of the label exceeds the range of the image, the detection accuracy of the YoloV5 network model will decrease. Therefore, when implementing the code, a certain distance needs to be reserved between the detection border and the image edge, and it is usually best to set it within the range of 1 - 100 pixels.

[0098] Step S230: Train on the constructed parking space dataset and the transformed training labels using the YoloV5 network model, and use the adjacent heterogeneous non - maximum suppression method during the model training process. According to the relevant solutions, there will be a misjudgment rate during the inference process of the model. For example, for the detection of the parking space border line, the same border line will change in category due to the rotation of the vehicle. If the accuracy of the model is not high enough, it may lead to the situation that a border line is judged as multiple types under the same confidence threshold, resulting in misjudgment. In order to improve the recognition accuracy of the model, in some embodiments, the adjacent heterogeneous non - maximum suppression method is applied to the training process of the YoloV5 network model to improve the detection accuracy. Specifically:

[0099] Step S2301: Train according to the general process, set the Batch size of the YoloV5 network model to 300 and epochs to 50.

[0100] Step S2302: Cancel the two data augmentation processing operations of flipping and rotation.

[0101] Since the straight line detection of the rectangular box used in this method depends heavily on the orientation of the straight lines in the image, and some operations of data augmentation will change the orientation of the straight lines in the training data, such as flipping and rotation, etc., the above two data augmentation methods are directly cancelled. Additionally, it is also possible to recalculate the orientation of the parking space boundary lines after performing the above two data augmentation operations, but this method is relatively more complex and will not be elaborated here.

[0102] Step S2303: According to the spatial structure characteristics of the parking space, there is exactly one parking space boundary line within a certain area range. Iteratively calculate the parking space boundary lines E = {e n |n = 1, 2, …, m} detected in the image, and let the current parking space boundary line be e1.

[0103] Step S2304: Calculate the Euclidean distances between e1 and all the parking space boundary lines in E, and the calculation results form a new set D e = {d n |n = 1, 2, …, m}. In some embodiments, in addition to calculating the Euclidean distance, the KDtree (short for k-dimensional tree, which is a tree-shaped data structure for storing instance points in a k-dimensional space for fast retrieval) algorithm can also be used for calculation.

[0104] Step S2305: For the set D s = {d n |d n < d min , d n ∈ D e}, select the element with the highest credibility as the parking space boundary line at the position of this area, where d min is a user-defined threshold.

[0105] Step S240: Perform conditional constraints according to the characteristics of the spatial structure of the parking space, select boundary lines for a single parking space recognized by the model, and combine them to obtain a complete parking space. This step adds constraint conditions for the characteristics of the parking space to extract the position of the parking space in space, can identify and give the spatial position of the parking space of any quadrilateral, and has strong robustness to the interference of different lighting and ground conditions.

[0106] Step S240 mainly uses the two boundary lines and their endpoints selected for a single parking space for combined connection to construct a complete parking space of any quadrilateral, without considering the angle of the parking space, which greatly improves the generality. Specifically:

[0107] Step S2401: Iteratively calculate the parking spaces P = {p n |n = 1, 2, …, m} detected in the image, and let the current parking space be p1.

[0108] Step S2402: Calculate the Euclidean distance between p1 and the midline of E, and the calculation results form a new set D p ={d n |n = 1, 2, …, m}.

[0109] Step S2403: Sort D p in ascending order, and the sorting results form set D g ={k n |n = 1, 2, …, m}, select the sidelines e p and e k1 corresponding to k1 and k2 (i.e., the two smallest elements in D k2 ) as the two sidelines on both sides of p1. If both e k1 and e k2 are less than D min , then a parking space can be formed; otherwise, exclude the possibility that this position is a parking space.

[0110] Step S2404: Assume e k1 ={p o1 , p i2}, e k2 ={p o3 , p i4}, then it can be deduced that the sideline at the entrance of the parking space is q1 = {p o1 , p i3}, and the bottom sideline at the inner side is q2 = {p o2 , p i4}. At this time, the set S = {e k1 , e k2 , q1, q2} is a complete four-sided parking space, as shown in Figure 8 .

[0111] The beneficial effects of the parking space positioning method provided in this embodiment are as follows:

[0112] (1) Related solutions use a combination of key point detection technology and traditional vision line detection technology for detection, resulting in low spatial correlation between points and requiring cumbersome operations to obtain a complete parking space border. When generating labels in this method, the parking space sidelines perpendicular or parallel to the bottom edge of the image are attached to the midpoints of the opposite sides of the detection border, and the parking space sidelines at the remaining angles to the bottom edge of the image are attached to the diagonals of the detection border. Then, the deep neural network is used to directly detect the parking space sidelines, retaining the spatial relationship of the parking space corner points and improving the calculation efficiency.

[0113] (2) Related solutions use template or fixed-angle parking space matching, resulting in a decrease in versatility. This method uses a deep neural network to detect parking spaces that can form any quadrilateral, greatly improving the versatility.

[0114] (3) The fault tolerance of this method is high. As Figure 6 shown, the rightmost origin in the right figure indicates that the detection model judges this as a parking space (false detection). However, the processing flow of this method acts on the training process of the model through the adjacent heterogeneous non-maximum suppression method, excluding the possibility that this position is a parking space, so the parking space border is not recognized and marked.

[0115] (4) This method completely uses a deep neural network for recognition in the detection of sidelines and parking spaces. Therefore, when the amount of data is sufficient, it has strong anti-interference ability against factors such as environmental light and ground stains.

[0116] (5) Compared with the existing parking space key point detection scheme, after being processed in step S240, this method can construct a complete parking space through simple calculations without the need for line detection. On the premise of ensuring high accuracy and low omission rate, the performance requirements for computing devices are relatively low, and the detection frame rate using NVIDIA jetson nano reaches about 30 FPS.

[0117] Referring to Figure 10 , an embodiment of the present invention provides a parking space positioning system, including a data annotation unit 100, a label generation unit 200, a model training unit 300, and a parking space generation unit 400, where:

[0118] The data annotation unit 100 is used to obtain the parking space area in the parking space image, and annotate the entrance corner points of the parking space, the interior corner points and the midpoint of the parking space area. The interior corner points of the parking space area refer to the inner corner points of the parking space or the endpoints of the sidelines of the parking space area close to the inner corner points of the parking space.

[0119] The label generation unit 200 is used to generate training labels according to the annotated entrance corner points of the parking space, the interior corner points and the midpoint of the parking space area. Among them, the midpoint of the parking space area is set at the midpoint of the detection border, and the corresponding parking space sidelines are set on the diagonal or the midline of the detection border according to the coordinate transformation of the entrance corner points of the parking space and the interior corner points of the parking space area.

[0120] The model training unit 300 is used to train a deep convolutional neural network model through the annotated parking space images and training labels to obtain the recognition result output by the deep convolutional neural network model.

[0121] The parking space generation unit 400 is used to generate a parking space with complete four sides according to the recognition result.

[0122] It should be noted that the system embodiment of the present invention and the above method embodiment are based on the same inventive concept. Therefore, the relevant content of the above method embodiment is equally applicable to the system embodiment of the present invention, and will not be elaborated here.

[0123] An embodiment of the present invention provides an unmanned sanitation robot. Specifically, the unmanned sanitation robot includes one or more control processors and a memory. In this example, one control processor is taken as an example. The control processor and the memory can be connected through a bus or other means. In this example, the connection through the bus is taken as an example.

[0124] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The control processor realizes the parking space positioning method in the above method embodiment by running the non-transitory software programs, instructions, and modules stored in the memory. The memory can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function. In addition, the memory can include a high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely set relative to the control processor, and these remote memories can be connected to the unmanned sanitation robot through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. The one or more modules are stored in the memory and, when executed by the one or more control processors, execute the parking space positioning method in the above method embodiment. For example, execute the method steps S110 to S140 described above in Figure 5 or the method steps S210 to S240 in Figure 9

[0125] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more control processors, for example, it can cause the one or more control processors to execute the parking space positioning method in the above method embodiment. For example, execute the method steps S110 to S140 described above in Figure 5 or the method steps S210 to S240 in Figure 9

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform. Those skilled in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the method embodiments as described above. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0128] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A parking space positioning method, characterized in that, Including the following steps: Obtain the parking space area in the parking space image, and mark the parking space entrance corner points, the interior corner points and the midpoint of the parking space area. The interior corner points of the parking space area refer to the inner corner points of the parking space or the endpoints of the side line of the parking space area close to the inner corner points of the parking space; Generate training labels according to the marked parking space entrance corner points, the interior corner points and the midpoint of the parking space area. Among them, the generation of training labels is to generate a detection border and a label for generating the detection border in the parking space image, and one of the generation of training labels is to set the midpoint of the parking space area at the midpoint of the detection border, and the other is to set the corresponding parking space side line on the diagonal or the midline of the detection border according to the coordinate transformation of the parking space entrance corner points and the interior corner points of the parking space area; Train a deep convolutional neural network model through the marked parking space image and the training labels to obtain the recognition result output by the deep convolutional neural network model; Generate a parking space with a complete four sides according to the recognition result.

2. The parking space positioning method according to claim 1, wherein The setting the parking space side line on the diagonal or the midline of the detection border according to the coordinate transformation of the parking space entrance corner points and the interior corner points of the parking space area includes: Calculate the angle between the corresponding parking space side line and the bottom edge of the parking space image according to the parking space entrance corner points and the interior corner points of the parking space area; When the parking space side line is perpendicular or parallel to the bottom edge of the parking space image, attach the parking space side line to the midpoint of the opposite side of the detection border through coordinate transformation; when the parking space side line is neither perpendicular nor parallel to the bottom edge of the parking space image, attach the parking space side line to the diagonal of the detection border through coordinate transformation.

3. The parking space positioning method according to claim 2, wherein The deep convolutional neural network model is the YoloV5 network model.

4. The parking space positioning method according to claim 2, wherein The training the deep convolutional neural network model through the marked parking space image and the training labels includes: Obtain all recognized parking space side lines; Calculate the Euclidean distance between the current parking space boundary line and each of the remaining parking space boundary lines to obtain a set , where represents the Euclidean distance between the current parking space boundary line and one of the remaining parking space boundary lines, represents the number of parking space boundary lines; Select the element with the highest credibility from the set as the parking space boundary corresponding to a regional position in the parking space image, where is a custom threshold value.

5. The parking space positioning method according to claim 4, wherein The generating a parking space with a complete four sides according to the recognition result includes: Obtain all recognized parking spaces; Calculate the Euclidean distance between the current parking space and each recognized parking space side line to obtain a distance set; Select two parking space side lines corresponding to the two smallest Euclidean distances in the distance set; When both of the two parking space side lines are less than a preset value, combine the two parking space side lines and the corresponding endpoints of the two parking space side lines to obtain the four side lines of the current parking space.

6. The parking space positioning method according to claim 3, wherein, During the training process of the deep convolutional neural network model, delete the flipping and rotating processing of the parking space image.

7. The parking space positioning method according to any one of claims 1 to 6, characterized in that, Before the obtaining the parking space area in the parking space image, it further includes: Fuse the images taken by the four camera devices in the front, back, left and right of the driverless sanitation robot to obtain the parking space image.

8. A parking space positioning system, characterized in that, Including: A data annotation unit for obtaining the parking space area in the parking space image, and marking the parking space entrance corner points, the interior corner points and the midpoint of the parking space area. The interior corner points of the parking space area refer to the inner corner points of the parking space or the endpoints of the side line of the parking space area close to the inner corner points of the parking space; A label generation unit, configured to generate training labels according to the marked corner points of the parking space entrance, the interior corner points and the midpoint of the parking space area, wherein the generation of the training labels is to generate a detection border and a label for the detection border in the parking space image, and one of the generation of the training labels is to set the midpoint of the parking space area at the midpoint of the detection border, and the other is to set the corresponding parking space side line on the diagonal or the midline of the detection border according to the coordinate transformation of the parking space entrance corner point and the interior corner points of the parking space area; A model training unit, configured to train a deep convolutional neural network model through the marked parking space image and the training labels, and obtain the recognition result output by the deep convolutional neural network model; A parking space generation unit, configured to generate a parking space with complete four sides according to the recognition result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the parking space positioning method according to any one of claims 1 to 7.

10. An unmanned sanitation robot, characterized in that: Comprising at least one control processor and a memory for communicatively connecting with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the parking space positioning method according to any one of claims 1 to 7.

Images