Wheel stopper rod marking method, device, computer equipment and storage medium
By identifying and mapping the wheelbarrow around the vehicle and performing maximum actual distance calculation, the problem of difficulty in labeling in the existing technology is solved, and the accuracy and efficiency of automatic parking are improved.
Patent Information
- Application Number
- CN202410751052.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-06-12
AI Technical Summary
The prior art is difficult to effectively mark the wheelbarrow around the vehicle, which affects the accuracy and efficiency of automatic parking technology.
By obtaining the vehicle's target map and image sequence, identifying and mapping the parking space and wheelbarrow rods, performing nearest neighbor matching and maximum actual distance calculations, and automatically marking the position and distance of the wheelbarrow rods to reduce the cost of human labeling.
Accurate and automatic labeling of the wheelbarrow around the vehicle is achieved, improving the accuracy and efficiency of automatic parking, and reducing the labeling cost.
Smart Images

Figure CN118334662B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a wheel stopper marking method, device, computer equipment and storage medium. Background Art
[0002] With the continuous development of science and technology, automatic parking technology has become an increasingly popular assisted driving function in modern cars, which can assist vehicles to automatically find and park in parking spaces in complex environments. In automatic parking technology, it is necessary to calculate the maximum distance that a vehicle can park in a parking space, which requires marking the wheel stoppers in the parking spaces around the vehicle to determine the maximum distance. Therefore, how to mark the wheel stoppers in the parking spaces around the vehicle has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] In view of this, the present application provides a wheel stopper marking method, device, computer equipment and storage medium to solve the problem of how to mark the wheel stoppers in the parking spaces around the vehicle.
[0004] In a first aspect, the present application provides a wheel blocking rod marking method, the method comprising:
[0005] Obtain a target map of the vehicle and an image sequence corresponding to the target map;
[0006] identifying a parking space and a wheel stop within each image in the image sequence;
[0007] Based on the coordinate mapping relationship between the pre-acquired image and the map, the parking space and the wheel stopper in each image are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map;
[0008] Performing a nearest neighbor matching operation on each map parking space and map wheel stopper to determine the map wheel stopper in each map parking space in the target map;
[0009] For each mapped parking space, determine the maximum actual distance between each pixel point of the wheel stopper in the mapped parking space and the target straight line, where the target straight line is a straight line perpendicular to the long side of the mapped parking space and where at least one corner point of the entrance of the mapped parking space is located;
[0010] For each map parking space, the maximum actual distances of all determined pixel points are sorted, and a target maximum actual distance is selected from them based on the sorting. The target actual maximum distance is the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold. The target maximum actual distance is used as the marking result of the map wheel stopper in the map parking space, and the marking results of each map wheel stopper in the target map are obtained.
[0011] In the above technical solution, based on the parking spaces and wheel stops identified in the target map corresponding image sequence and the coordinate mapping relationship between the image and the map, each map parking space and map wheel stop in the target map can be identified, and the nearest neighbor matching operation is performed to obtain the map wheel stop in each map parking space, so as to calculate the maximum actual distance from each pixel point of the map wheel stop in each map parking space to the parking space entrance, so as to select the larger of the two maximum actual distances that meet the difference between the maximum actual distances of two adjacent pixel points greater than or equal to the first threshold from the sorted maximum actual distances, thereby obtaining the annotation result of the map wheel stop in the map parking space. Automatic annotation of the wheel stop in the parking space around the vehicle is realized, and manual annotation is not required to reduce the annotation cost. The result of automatic annotation is the target maximum actual distance from the map wheel stop to the vehicle entrance, that is, the maximum distance that the vehicle can park in the middle parking space, which is convenient for subsequent rapid automatic parking.
[0012] In some optional embodiments, identifying a parking space and a wheel stopper in each image in the image sequence includes:
[0013] Determine a plurality of wheel blocking rod partial images and a plurality of parking space partial images in the image sequence, wherein the coordinate mapping relationship between the image and the map includes a coordinate inverse mapping relationship between the map and the bird's-eye view and a coordinate inverse mapping relationship between the image and the bird's-eye view;
[0014] Based on the coordinate mapping relationship between the pre-acquired image and the map, the parking space and the wheel stopper in each image are mapped to the target map respectively, and each map parking space and map wheel stopper in the target map are obtained, including:
[0015] The images collected at the same time in the image sequence are stitched into a bird's-eye view to obtain the bird's-eye view of each frame corresponding to the target map;
[0016] Based on the inverse mapping relationship between the coordinates of the map and the bird's-eye view, the corresponding coordinates of each pixel point in the target map in each frame of the bird's-eye view are determined to obtain a set of bird's-eye view coordinates corresponding to each pixel point in the target map;
[0017] For each bird's-eye view coordinate set, based on the coordinate inverse mapping relationship between the image and the bird's-eye view, the coordinates corresponding to each bird's-eye view coordinate in the bird's-eye view coordinate set in each image are determined to obtain an image coordinate set corresponding to the bird's-eye view coordinate set;
[0018] According to the coordinates in each image coordinate set, the coordinates of each pixel point in each wheel stopper partial image, and the coordinates of each pixel point in each parking space partial image, each wheel stopper partial image and each parking space partial image are respectively mapped to each frame of the bird's-eye view image to obtain the wheel stopper and the parking space in each frame of the bird's-eye view;
[0019] According to the coordinates in each bird's-eye view coordinate set and the coordinates of each pixel point in each frame of the bird's-eye view, the wheel stopper and the parking space in each frame of the bird's-eye view are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map.
[0020] In the above technical solution, the parking spaces and wheel stoppers that can be quickly and accurately determined from the image sequence can be mapped to the target map by coordinate mapping, so as to quickly and accurately determine the map parking spaces and map wheel stoppers in the target map, and avoid the low accuracy and efficiency caused by the operation of identifying and classifying parking spaces and wheel stoppers on an entire target map.
[0021] In some optional embodiments, each pixel point in each wheel stopper partial image corresponds to a wheel stopper weight, and each pixel point in each parking space partial image corresponds to a parking space weight. According to the coordinates in each image coordinate set, the coordinates of each pixel point in each wheel stopper partial image, and the coordinates of each pixel point in each parking space partial image, each wheel stopper partial image and parking space partial image are respectively mapped to each frame of the bird's-eye view image to obtain the wheel stopper and parking space in each frame of the bird's-eye view image, including:
[0022] According to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod local image, each wheel blocking rod local image is converted into a heat map to obtain multiple wheel blocking rod heat maps;
[0023] According to the coordinates in each image coordinate set, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal values of each pixel point in each wheel stopper rod thermal map, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image coordinate set in the image to which it belongs, and obtain the parking space weight and thermal value corresponding to each image coordinate;
[0024] For each image coordinate in each image coordinate set, the parking space weight and thermal value corresponding to the image coordinate are respectively integrated into one parking space weight and one thermal value, and the integrated parking space weight and thermal value are respectively used as the parking space weight and thermal value of the pixel point corresponding to the image coordinate in its corresponding bird's-eye view;
[0025] Based on the thermal value of each pixel point in each frame of the bird's-eye view, the thermal area of the wheel blocking rod in each frame of the bird's-eye view is determined respectively;
[0026] The wheel blocking rod thermal area in each frame of the bird's-eye view is divided into the wheel blocking rod connected area to obtain each wheel blocking rod in each frame of the bird's-eye view;
[0027] An image composed of pixel points whose parking space weights are greater than or equal to the second threshold in each frame of the bird's-eye view image is determined as a parking space.
[0028] In the above technical scheme, the granularity of the division of the wheel stopper connected area of the wheel stopper heat map is controlled at the level of the bird's-eye view with higher accuracy and relatively larger data volume, so that the divided wheel stopper can be subsequently mapped to the target map, so as to quickly and accurately obtain the map wheel stopper in the target map, avoid dividing the wheel stopper in the target map with a larger field of view and prone to errors, improve the accuracy of wheel stopper recognition in the map, and avoid dividing the wheel stopper connected area for each image in the image sequence, thereby improving the efficiency of wheel stopper recognition.
[0029] In some optional embodiments, the method further comprises:
[0030] Receive a request for marking the wheel stopper in the target bird's-eye view based on the target map, and obtain the target bird's-eye view corresponding to the target map;
[0031] Based on the coordinate mapping relationship between the pre-acquired map and the bird's-eye view, the map parking space and the map wheel stopper predetermined in the target map are mapped to the target bird's-eye view to obtain the parking space and the wheel stopper in the target bird's-eye view;
[0032] Perform a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view;
[0033] For each parking space in the target bird's-eye view, determine the maximum actual distance between each pixel point of its inner wheel stopper and the target straight line;
[0034] For each parking space in the target bird's-eye view, the maximum actual distances of all the determined pixel points are sorted, and the target maximum actual distance is selected based on the sorting. The target maximum actual distance is the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold. The target maximum actual distance is used as the labeling result of the wheel stopper in the parking space, and the labeling results of each wheel stopper in the target bird's-eye view are obtained.
[0035] In the above technical solution, when it is necessary to mark the parking spaces and wheel blocks in the bird's-eye view, the pre-marked map road blocks and map parking spaces in the target map can be mapped back to the corresponding bird's-eye view based on the coordinate mapping relationship between the map and the bird's-eye view, so as to facilitate the subsequent determination of the target maximum actual distance of the parking spaces and wheel blocks in the bird's-eye view. There is no need to manually mark the parking spaces and wheel blocks in the bird's-eye view, which can avoid the situation of human omission and mismarking and improve the accuracy of marking. In addition, if it is necessary to mark multiple bird's-eye views, the pre-marked map road blocks and map parking spaces can be mapped to the corresponding positions in multiple bird's-eye views at one time through the target map for subsequent calculations, avoiding marking each bird's-eye view, reducing the data of repeated marking, and thus improving the marking efficiency of the bird's-eye view.
[0036] In some optional embodiments, the method further comprises:
[0037] Receive a request to mark the wheel stopper in the target bird's-eye view, and obtain the target bird's-eye view corresponding to the target map;
[0038] Based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view, the wheel blocking rods and parking spaces pre-identified in the multiple images corresponding to the target bird's-eye view are mapped into the target bird's-eye view to obtain the parking spaces and wheel blocking rods in the target bird's-eye view;
[0039] Perform a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view;
[0040] For each parking space in the target bird's-eye view, determine the maximum actual distance between each pixel point of its inner wheel stopper and the target straight line;
[0041] For each parking space in the target bird's-eye view, the maximum actual distances of all the determined pixel points are sorted, and the target maximum actual distance is selected based on the sorting. The target maximum actual distance is the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold. The target maximum actual distance is used as the labeling result of the wheel stopper in the parking space, and the labeling results of each wheel stopper in the target bird's-eye view are obtained.
[0042] In the above technical scheme, by mapping the parking spaces and wheel stoppers pre-identified in each image corresponding to the target bird's-eye view into the target bird's-eye view, it is convenient to determine the wheel stoppers in the parking space in the target bird's-eye view, and select the target maximum actual distance from the maximum actual distance between each pixel point of the wheel stopper in the parking space and the target straight line to complete the labeling of the wheel stoppers in the target bird's-eye view, thereby realizing the labeling of each wheel stopper in the bird's-eye view, without the need to manually label the bird's-eye view, or to additionally train models to identify wheel stoppers and parking spaces in bird's-eye views with multiple perspectives and whose object shapes may change, thereby reducing the labeling cost and improving the labeling accuracy.
[0043] In some optional embodiments, identifying a parking space and a wheel stopper in each image in the image sequence includes:
[0044] Determine a plurality of wheel stopper partial images and a plurality of parking space partial images in the image sequence, each pixel point in each wheel stopper partial image corresponds to a wheel stopper weight, and each pixel point in each parking space partial image corresponds to a parking space weight;
[0045] Based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view, the wheel stopper and the parking space pre-identified in the multiple images corresponding to the target bird's-eye view are mapped to the target bird's-eye view to obtain the parking space and the wheel stopper in the target bird's-eye view, including:
[0046] According to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod local image, each wheel blocking rod local image is converted into a heat map to obtain multiple wheel blocking rod heat maps;
[0047] Based on the coordinate mapping relationship between the image and the bird's-eye view, the coordinates of each pixel point in the target bird's-eye view corresponding to each image corresponding to the target bird's-eye view are determined, and the image coordinate subset corresponding to each pixel point in the target bird's-eye view is obtained;
[0048] For each subset of image coordinates, according to the coordinates of each pixel point in each image corresponding to the target bird's-eye view, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal values of each pixel point in each wheel stopper thermal map, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image to which it belongs, and obtain the parking space weight and thermal value corresponding to each image coordinate;
[0049] For each image coordinate subset, the parking space weights and thermal values corresponding to each image coordinate in the image coordinate subset are respectively integrated into one parking space weight and one thermal value, and the integrated parking space weights and thermal values are respectively used as the parking space weights and thermal values of the corresponding pixel points of the image coordinate subset in the target bird's-eye view;
[0050] Determine the thermal area of the wheel blocking rod in the target bird's-eye view based on the thermal value of each pixel point in the target bird's-eye view;
[0051] The wheel blocking rod thermal area in the target bird's-eye view is divided into a wheel blocking rod connected area to obtain each wheel blocking rod in the target bird's-eye view;
[0052] An image composed of pixel points whose parking space weights in the target bird's-eye view are greater than or equal to the second threshold is determined as a parking space.
[0053] In the above technical solution, the recognition of parking spaces and wheel stoppers in the target bird's-eye view is achieved by mapping the parking spaces and wheel stoppers pre-identified in each image corresponding to the target bird's-eye view into the target bird's-eye view. There is no need to additionally train a model to recognize bird's-eye views with multiple perspectives and possibly changing object shapes, thereby improving the accuracy of recognition of parking spaces and wheel stoppers in the bird's-eye view.
[0054] In some optional embodiments, obtaining a target map of a vehicle and an image sequence corresponding to the target map includes:
[0055] Obtain the image sequence captured by each camera on the vehicle and the semantic map of the vehicle;
[0056] Building a scene map including the ground on which the vehicle is traveling, the parking space on the ground, and the wheel stopper on the ground based on the image sequence;
[0057] Based on the coordinate mapping relationship between the image and the map, determine the corresponding coordinates of each pixel in the scene map in each image;
[0058] According to the coordinates of each pixel in the scene map in each image, as well as the coordinates and color values of the pixel in each image, the color value corresponding to each pixel in the scene map is determined to obtain a color scene map;
[0059] According to the annotation information of the scene around the vehicle in the semantic map, objects other than the ground, the parking spaces on the ground and the wheel blocking rods on the ground in the color scene map are deleted to obtain the target map.
[0060] In a second aspect, the present application provides a wheel blocking rod marking device, the device comprising:
[0061] An acquisition module, used to acquire a target map of the vehicle and an image sequence corresponding to the target map;
[0062] A recognition module, used for recognizing a parking space and a wheel stopper in each image in the image sequence;
[0063] A mapping module, for mapping the parking spaces and wheel stoppers in each image to the target map based on the coordinate mapping relationship between the pre-acquired image and the map, to obtain each map parking space and map wheel stopper in the target map;
[0064] A matching module, used for performing a nearest neighbor matching operation on each map parking space and the map wheel stopper, and determining the map wheel stopper in each map parking space in the target map;
[0065] A first determination module is used to determine, for each mapped parking space, the maximum actual distance between each pixel point of the wheel stopper in the mapped parking space and a target straight line, where the target straight line is a straight line away from the wheel stopper and perpendicular to the long side of the mapped parking space and where at least one corner point of the entrance of the mapped parking space is located;
[0066] The second determination module is used to sort the maximum actual distances of all determined pixel points for each map parking space, and select a target maximum actual distance based on the sorting, the target actual maximum distance being the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and use the target maximum actual distance as the marking result of the map wheel stopper in the map parking space to obtain the marking result of each map wheel stopper in the target map.
[0067] In a third aspect, the present application provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the wheel stop rod marking method of any embodiment of the first aspect by executing the computer instructions.
[0068] In a fourth aspect, the present application provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the wheel stop lever marking method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0070] Figure 1 is a schematic flow chart of a wheel blocking rod marking method according to an embodiment of the present application;
[0071] Figure 2 It is a schematic diagram of the maximum actual distance in the vertical map parking space;
[0072] Figure 3 It is a schematic diagram of the maximum actual distance in the oblique map parking space;
[0073] Figure 4 is a schematic flow chart of another wheel blocking rod marking method according to an embodiment of the present application;
[0074] Figure 5 is a flow chart of another wheel blocking rod marking method according to an embodiment of the present application;
[0075] Figure 6 is a schematic flow chart of another wheel blocking rod marking method according to an embodiment of the present application;
[0076] Figure 7 is a structural block diagram of a wheel blocking rod marking device according to an embodiment of the present application;
[0077] Figure 8 It is a schematic diagram of the hardware structure of the computer device of an embodiment of the present application. DETAILED DESCRIPTION
[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.
[0079] The embodiment of the present application provides a wheel stopper marking method, which maps the parking spaces and wheel stoppers in the automatically identified target image sequence to the map of the vehicle to obtain the parking spaces and wheel stoppers in the map, and then identifies the maximum distance between the wheel stoppers in each parking space in the map and the parking space entrance, thereby achieving the effect of identifying the wheel stoppers around the vehicle and the maximum distance that the vehicle can park in the middle parking space. The wheel stopper marking method of the present application is described in detail below.
[0080] According to an embodiment of the present application, an embodiment of a wheel block lever marking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0081] In an embodiment of the present application, a wheel stopper marking method is provided, which can be used in a computer device, such as a vehicle-mounted processor, a calculator, a server, or a device with data calculation and data analysis processing capabilities. Figure 1 is a flow chart of a wheel blocking rod marking method according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:
[0082] Step S101, obtaining a target map of the vehicle and an image sequence corresponding to the target map.
[0083] Through each camera on the vehicle, a series of images can be collected during the driving process of the vehicle to obtain an image sequence, and a scene map of the vehicle can be constructed based on the image sequence. Based on the coordinate mapping relationship between the image and the map, the coordinates of each pixel point on the scene map are found to correspond to the pixel points on each image in the image sequence. If the coordinates in the scene map directly correspond to the pixel points of a certain image in the image sequence, the RGB (Red Green Blue) value of the pixel point in the image is the RGB value corresponding to the map coordinate; if not, the RGB value of the coordinate corresponding to the coordinate in the image in the map is obtained by interpolation. The RGB value corresponding to each coordinate in the map is weighted summed in the map (the determination of the weight can be adjusted according to factors such as the camera's viewing angle and / or distance), and the RGB value of the pixel point indicated by each coordinate in the map is obtained, and a whole RGB scene map, that is, a color scene map, is obtained, and the non-ground objects of the color scene map are deleted in combination with the pre-acquired semantic map to obtain the target map. The specific method for obtaining the semantic map is a relatively mature technology at present, and will not be repeated in the embodiments of this application. The coordinate mapping relationship between the vehicle image and the map is specifically introduced in subsequent examples.
[0084] Each camera on the vehicle can be each camera in the surround camera system of the vehicle. Generally speaking, the general process of constructing a scene map of the vehicle based on an image sequence can be to select a reference image from the image sequence. Taking this image as a reference, relative motion estimation is performed on other images. Specifically, for images of other frames in the image sequence, computer vision feature matching methods such as feature point matching or optical flow method, or laser radar method, the motion relationship of the camera is calculated to obtain the camera pose change (rotation matrix R and translation vector T) from the reference image to the image; for images of other frames in the image sequence, the corresponding positions of all pixels in the image in the reference image are calculated according to the camera pose of the reference image and the obtained camera pose change, and the pixels in all other frames are projected onto the reference image to obtain the scene map. The specific relative motion estimation method, the computer vision feature matching method for calculating the camera pose change, the radar laser fee for calculating the camera pose change, the camera pose change calculation method, and the corresponding position and projection method of images of other frames in the reference image are currently relatively mature technologies and will not be repeated in the embodiments of this application.
[0085] Step S102: identifying a parking space and a wheel stopper in each image in the image sequence.
[0086] Each camera on the vehicle corresponds to a trained semantic segmentation model and a parking space detection model. The images collected by each camera in the image sequence are respectively input into the semantic segmentation model corresponding to the camera. The semantic segmentation model can identify and annotate the wheel stopper in the input image. The annotation result includes the wheel stopper weight corresponding to each pixel point on the wheel stopper identified in the input image. The wheel stopper weight is used to indicate the probability that the pixel point is a pixel point on the wheel stopper.
[0087] The images captured by each camera in the image sequence are input into the parking space detection model corresponding to the camera. The parking space detection model can identify and mark the parking spaces in the input images. The marking results include the four corner points of the parking space boundary identified in the input image and the parking space weights corresponding to the pixels on the identified parking spaces. The parking space weights are used to indicate the probability that the pixel is a pixel on the parking space. The specific marking method of the four corner points of the parking space boundary can be set by yourself. In one example, the corner point at the upper right corner of the parking space entrance is defined as point 1, and the remaining three corner points counterclockwise are point 2, point 3, and point 4, which are recorded as slot (p1, p2, p3, p4). In this way, the four corner points in the parking space are marked to facilitate the subsequent determination of the target straight line.
[0088] It should be noted that the semantic segmentation model can not only identify the wheel stopper in the input image, but also identify whether there is a wheel stopper in the input image and whether the wheel stopper is blocked. If necessary, it can also identify the specific type of the wheel stopper and mark the identified target. In the subsequent mapping process, the annotation results can be mapped to the bird's-eye view or the target map to intuitively understand whether the wheel stopper is blocked, etc., to assist the vehicle's automatic parking.
[0089] Step S103 , based on the coordinate mapping relationship between the pre-acquired image and the map, the parking space and the wheel stopper in each image are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map.
[0090] Based on the coordinate mapping relationship between the pre-acquired image and the map, each pixel point annotated with the annotation results output by the semantic segmentation model corresponding to each camera can be mapped to the target map respectively, and the annotation results mapped to the target map can be weighted. In addition, each pixel point annotated with the annotation results output by the parking space detection model corresponding to each camera can be mapped to the target map respectively, and the annotation results mapped to the target map can be weighted. Finally, the annotated map parking space and map wheel stopper can be obtained on the target map. The specific weighting scheme in the target map should be related to the distance from the center point of the vehicle's parking space to the coordinate in the target map, for example, the inverse of the distance can be used as the weight for weighting.
[0091] Step S104, performing a nearest neighbor matching operation on each mapped parking space and the mapped wheel stopper to determine the mapped wheel stopper in each mapped parking space in the target map.
[0092] Traverse the map parking spaces in the target map, traverse the map wheel stop for each map parking space, calculate the distance between the map parking space and the map wheel stop, and check whether the distance is less than or equal to a preset threshold. If so, determine that there is a matching relationship between the map parking space and the map wheel stop, record the matching result and continue to look for the next matching relationship; if not, continue to calculate the distance between the map parking space and the next map wheel stop until each map parking space is traversed, thereby determining the map wheel stop in each map parking space.
[0093] Step S105: for each mapped parking space, determine the maximum actual distance between each pixel point of the wheel stopper on the mapped parking space and the target straight line.
[0094] The target straight line is a straight line that is away from the wheel stop on the map, perpendicular to the long side of the map parking space, and where at least one corner point of the entrance of the map parking space is located. For each pixel point of the wheel stop on the map in the map parking space, its vertical distance from the target straight line in the target map can be calculated, and the vertical distance can be converted into the distance in the actual scene and the converted distance can be determined as the maximum actual distance between the pixel point and the target straight line, thus obtaining the maximum actual distance between each pixel point of the wheel stop on the map parking space and the target straight line. In one example, for a vertical map parking space, the target straight line can be as follows: Figure 2 As shown by the straight line 210 in the figure, the maximum actual distance between a pixel point d1 of the wheel stopper on the map and the target can be expressed as Figure 2 As shown in the distance d210 in FIG. 1 ; for a diagonal map parking space, the target straight line can be as follows Figure 3 As shown by the straight line 310 in the figure, the maximum actual distance between a pixel point d1 of the wheel stopper on the map and the target can be expressed as Figure 3 As shown in the distance d310.
[0095] Step S106, for each map parking space, sort the maximum actual distances of all determined pixel points, and select the target maximum actual distance based on the sorting, the target actual maximum distance being the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold, and use the target maximum actual distance as the marking result of the map wheel stop in the map parking space to obtain the marking result of each map wheel stop in the target map.
[0096] Among them, the first threshold can be set by itself. In the embodiment of the present application, the first threshold is taken as 5% of the smaller of the two maximum actual distances, and for each map parking space, the maximum actual distances between each pixel point on the map wheel stopper in the map parking space and the target straight line are sorted in order from small to large, and from the sorted maximum actual distances, the difference in the maximum actual distance between two adjacent pixel points is found to be greater than or equal to the first threshold in the direction of small to large, and the larger of the two maximum actual distances found is used as the target maximum actual distance, and the target actual maximum distance is used as the annotation result of the map wheel stopper in the map parking space, thereby obtaining the annotation results of each map wheel stopper in the target map.
[0097] In the embodiment of the present application, based on the parking spaces and wheel stops identified in the target map corresponding image sequence and the coordinate mapping relationship between the image and the map, each map parking space and map wheel stop in the target map can be identified, and the nearest neighbor matching operation is performed to obtain the map wheel stop in each map parking space, so as to calculate the maximum actual distance from each pixel point of the map wheel stop in each map parking space to the parking space entrance, so as to select the larger of the two maximum actual distances that meet the difference between the maximum actual distances of two adjacent pixel points greater than or equal to the first threshold from the sorted maximum actual distances, thereby obtaining the labeling result of the map wheel stop in the map parking space. Automatic labeling of the wheel stop in the parking space around the vehicle is achieved, without the need for manual labeling to reduce the labeling cost. The result of automatic labeling is the target maximum actual distance from the map wheel stop to the vehicle entrance, that is, the maximum distance that the vehicle can park in the middle parking space, which is convenient for subsequent rapid automatic parking.
[0098] In an embodiment of the present application, another wheel stopper marking method is provided, which can be used in a computer device, such as a vehicle-mounted processor, a calculator, a server, or a device with data calculation and data analysis processing capabilities. Figure 4 is a flow chart of another wheel blocking rod marking method according to an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:
[0099] Step S401, obtaining a target map of the vehicle and an image sequence corresponding to the target map.
[0100] Optionally, the above step S401 may include the following steps A1 to A5:
[0101] Step A1, obtaining the image sequence captured by each camera on the vehicle and the semantic map of the vehicle.
[0102] Step A2: constructing a scene map including the ground on which the vehicle is traveling, the parking space on the ground, and the wheel stopper on the ground based on the image sequence.
[0103] Step A3, based on the coordinate mapping relationship between the image and the map, determine the corresponding coordinates of each pixel point in the scene map in each image.
[0104] Step A4, according to the coordinates of each pixel in the scene map in each image, as well as the coordinates and color values of the pixel in each image, the color value corresponding to each pixel in the scene map is determined to obtain a color scene map.
[0105] Step A5, according to the annotation information of the scene around the vehicle in the semantic map, delete the objects except the ground, the parking space on the ground and the wheel blocking rod on the ground in the color scene map to obtain the target map.
[0106] For details of steps A1 to A5, see Figure 1 The step S101 is not described in detail here. It can be understood that ground objects include the ground, parking spaces on the ground and wheel stoppers on the ground, and non-ground objects may include pedestrians and objects such as signboards that are not needed when marking wheel stoppers.
[0107] A scene map is constructed based on the image sequence, and based on the coordinate mapping relationship between the image and the map, the corresponding coordinates of each pixel in the scene map in each image are determined to determine the color value of each pixel in the scene map, so as to obtain a target map by deleting objects except the ground, parking spaces on the ground and wheel stoppers on the ground in the color scene map in combination with the semantic map, thereby deleting obstacles that hinder annotation in the map, which is convenient for improving the accuracy and efficiency of annotating wheel stoppers in the target map in the future.
[0108] Step S402: determining a plurality of wheel blocking rod partial images and a plurality of parking space partial images in the image sequence.
[0109] Specifically, the images captured by each camera in the image sequence are respectively input into the semantic segmentation model corresponding to the camera. The semantic segmentation model can identify and mark the wheel stopper to output a local map of the wheel stopper. A similar parking space detection model can identify and mark the parking space based on the input image to output a local map of the parking space. In this way, multiple local maps of the wheel stopper and multiple local maps of the parking space are obtained in the image sequence.
[0110] The coordinate mapping relationship between the image and the map includes the inverse mapping relationship between the map and the bird's-eye view and the inverse mapping relationship between the image and the bird's-eye view. Specifically, the coordinate mapping relationship between the image and the map is used to indicate that for any point with coordinates pm on the map (the map includes the target map and the scene map), mapping it to the bird's-eye view of the tth frame (i.e., time t) can obtain the corresponding coordinates pcamt. The specific mapping process is as follows:
[0111] (1) Map pm to the coordinate system of the ground where the vehicle is located (i.e., the ground coordinate system): pbt=Tgtw×pm; where the entire process of (1) is used to indicate the inverse mapping relationship between the map and the bird's-eye view, pbt is the coordinate of the point on the map in the ground coordinate system, Tgtw represents the transformation matrix from the ground coordinate system to the world coordinate system at time t, which is the inverse mapping of Twgt, Twgt=Twe0×Te0et×Tetgt, Twgt represents the transformation matrix from the world coordinate system to the ground coordinate system at time t, Twe0 represents the transformation matrix from the world coordinate system to the vehicle coordinate system at time 0, Te0et represents the transformation matrix from time 0 to time t; pbt and pm both represent coordinates in meters.
[0112] (2) Map the points on the ground coordinate system to the bird's eye view (BEV): pcamt = mapping2(pbt), where pcamt is the coordinate corresponding to pbt on the image at time t, and the unit of the coordinate indicated by pcamt is pixel. Mapping2 represents the inverse mapping relationship between the image and the bird's eye view.
[0113] The inverse mapping relationship between the image and the bird's-eye view is used to indicate that for a point with coordinates pcam on an image of the t-th frame in the image sequence, it is mapped to the t-th frame of the bird's-eye view to obtain the corresponding coordinates pb, assuming that the coordinates of the point in the camera coordinate system of the corresponding camera in the image are pc=(x, y, z) T , the specific mapping process is as follows:
[0114] (1) Convert pcam to the camera coordinate system: pcam=K(x / z,y / z,1) T [:2], where K is the intrinsic parameter of the camera corresponding to the image, T represents transposition, and [:2] represents K(x / z,y / z,1) T The first two digits.
[0115] (2) The point in the camera coordinate system is first transferred to the vehicle center coordinate system, and then to the ground coordinate system: pg = Tge × Tec × pc, where pg represents the coordinates of the point on the image in the ground coordinate system, Tge represents the transformation matrix from the ground coordinate system to the vehicle coordinate system, and Tec represents the transformation matrix from the vehicle coordinate system to the vehicle center coordinate system.
[0116] (3) Transfer the coordinates of the points in (2) to the bird's-eye view map: pb = pg / lp × n = (_, _, 0, 1), where pb [1] = -pb [1], n represents the number of pixels in the bird's-eye view map, lp represents the distance of each pixel in the bird's-eye view map, and in the embodiment of the present application, the pixels of the bird's-eye view map are n = w = h = 1600, lp = 0.05m, w represents the width of the bird's-eye view map, and h represents the height of the bird's-eye view map. The first two digits in (_, _, 0, 1) represent the coordinate values to be calculated.
[0117] (4) Based on the three equations of pcam, pg and pb, the values of the three unknown variables x, y and z are solved to obtain pb, and then the corresponding coordinates of a point in the bird's-eye view of the tth frame on a certain image at time t in the image sequence are obtained.
[0118] Step S403 , based on the coordinate mapping relationship between the pre-acquired image and the map, the parking space and the wheel stopper in each image are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map.
[0119] In order to speed up the recognition efficiency of the map parking space and the map wheel stopper in the map, the above step S403 includes the following steps S4031 to S4035:
[0120] Step S4031, stitching the images collected at the same time in the image sequence into a bird's-eye view, and obtaining the bird's-eye view frames corresponding to the target map.
[0121] For the images captured by different cameras at the same time in the image sequence, they can be spliced into a bird's-eye view to obtain a frame of bird's-eye view corresponding to the target map at that time. In this way, the images captured by different cameras at each time can be spliced to obtain each frame of bird's-eye view corresponding to the target map. The specific method of converting multiple images into a bird's-eye view is a relatively mature technology at present, and the embodiments of this application will not be described in detail.
[0122] It should be noted that in order to ensure the accuracy of the stitching results, before stitching the images collected at the same time in the image sequence into a bird's-eye view, each image in the image sequence can be subjected to preprocessing operations such as denoising, color correction and brightness adjustment to improve the image quality.
[0123] Step S4032, based on the coordinate inverse mapping relationship between the map and the bird's-eye view, determine the corresponding coordinates of each pixel point in the target map in each frame of the bird's-eye view, and obtain a set of bird's-eye view coordinates corresponding to each pixel point in the target map.
[0124] For the coordinates of each pixel point in the target map, based on the inverse mapping relationship between the map and the bird's-eye view, the coordinates of the pixel point in each frame of the bird's-eye view corresponding to the target map can be found, thus obtaining a set of bird's-eye view coordinates corresponding to each pixel point in the target map.
[0125] Step S4033: for each bird's-eye view coordinate set, based on the coordinate inverse mapping relationship between the image and the bird's-eye view, determine the coordinates corresponding to each bird's-eye view coordinate set in each image, and obtain the image coordinate set corresponding to the bird's-eye view coordinate set.
[0126] For the bird's-eye view coordinates corresponding to a pixel point in the target map on each frame of the bird's-eye view, based on the inverse mapping relationship between the image and the bird's-eye view coordinates, the coordinates corresponding to the bird's-eye view coordinates can be found from each image corresponding to the bird's-eye view, and then the image coordinate set corresponding to each bird's-eye view coordinate set can be obtained.
[0127] Step S4034, according to the coordinates in each image coordinate set, the coordinates of each pixel point in each wheel stop rod partial image, and the coordinates of each pixel point in each parking space partial image, each wheel stop rod partial image and each parking space partial image are mapped to each frame of the bird's-eye view image, so as to obtain the wheel stop rod and parking space in each frame of the bird's-eye view.
[0128] The coordinates of each wheel stopper partial image and parking space partial image are consistent with the coordinates of the corresponding position in the corresponding image. Therefore, based on the coordinate inverse mapping relationship between the image and the bird's-eye view, the image coordinates corresponding to each bird's-eye view coordinate in the bird's-eye view coordinate set can be determined. The image coordinates corresponding to each bird's-eye view coordinate in each wheel stopper partial image and parking space partial image are mapped to the bird's-eye view to which the bird's-eye view coordinate belongs. The annotation results are weighted (the specific weight can be set by yourself). Finally, the pixel points labeled in each frame of the bird's-eye view can be obtained, and the wheel stopper and parking space labeled in each frame of the bird's-eye view can be obtained.
[0129] Step S4035, according to the coordinates in each bird's-eye view coordinate set and the coordinates of each pixel point in each frame of the bird's-eye view, the wheel stopper and the parking space in each frame of the bird's-eye view are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map.
[0130] The annotation results of the pixel points indicated by the corresponding bird's-eye view coordinates of each pixel point in the target map in each frame of the bird's-eye view are mapped to the target map, and each annotation result mapped to the target map is weighted (the specific weight should be related to the distance from the center point of the vehicle's parking space to the coordinates of the pixel point in the target map, for example, the inverse of the distance is used as the weight). Finally, the annotated pixel points in the target map can be obtained, and each map parking space and map wheel stop in the target map can be obtained.
[0131] The parking spaces and wheel stoppers that can be quickly and accurately determined from the image sequence are mapped to the target map by coordinate mapping, so as to quickly and accurately determine the map parking spaces and map wheel stoppers in the target map, and avoid the low accuracy and efficiency caused by the operation of identifying and classifying parking spaces and wheel stoppers on the entire target map.
[0132] In order to improve the accuracy of identifying the wheel stop lever in the map, the above step S4034 may include the following steps B1 to B6:
[0133] Step B1, converting each wheel blocking rod partial image into a heat map according to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod partial image, to obtain a plurality of wheel blocking rod heat maps.
[0134] Among them, each pixel point in each wheel stopper local image corresponds to a wheel stopper weight, and each pixel point in each parking space local image corresponds to a parking space weight. When the wheel stopper of each image in the image sequence is identified and annotated using the semantic segmentation model corresponding to each camera, the semantic segmentation model will not only output the local wheel stopper image, but also predict and output the wheel stopper heat map based on the wheel stopper weight corresponding to each pixel point in the local wheel stopper image. Specifically, for each wheel stopper local image, the semantic segmentation model that outputs the wheel stopper local image will map the pixel points in the wheel stopper local image whose corresponding wheel stopper weight is greater than a certain threshold to a specified color or grayscale level, so that the wheel stopper heat map can be obtained.
[0135] Step B2, based on the coordinates in each image coordinate set, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal values of each pixel point in each wheel stop rod thermal map, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image coordinate set in the image to which it belongs, and obtain the parking space weight and thermal value corresponding to each image coordinate.
[0136] The coordinates of each wheel stopper heat map are consistent with the coordinates of its corresponding wheel stopper local map. Therefore, based on the inverse mapping relationship between the image and the bird's-eye view, when the coordinates of each bird's-eye view coordinate in the bird's-eye view coordinate set are determined in each image, the image coordinates corresponding to each bird's-eye view coordinate in each wheel stopper heat map and the parking space local map can be clarified. The thermal value of the image coordinate indicating the pixel point corresponding to the bird's-eye view coordinate in each wheel stopper heat map is determined as the thermal value of the image coordinate in the wheel stopper heat map. If the image coordinate in a wheel stopper heat map does not correspond to a certain pixel point, the thermal value corresponding to the image coordinate can be calculated by interpolation, so as to obtain the thermal value corresponding to each image coordinate. The same operation is performed on each parking space local map to obtain the parking space weight corresponding to each image coordinate.
[0137] Step B3, for each image coordinate in each image coordinate set, the parking space weight and thermal value corresponding to the image coordinate are respectively integrated into one parking space weight and one thermal value, and the integrated parking space weight and thermal value are respectively used as the parking space weight and thermal value of the corresponding pixel point of the image coordinate in its corresponding bird's-eye view.
[0138] For the bird's-eye view coordinates corresponding to a pixel point in the target map on each frame of the bird's-eye view, the thermal value and parking space weight of each image coordinate corresponding to the bird's-eye view coordinate are mapped to the bird's-eye view, and the mapped thermal value and parking space weight are weighted respectively to obtain an integrated parking space weight and thermal value, so as to obtain the parking space weight and thermal value of the pixel point corresponding to the image coordinate in its corresponding bird's-eye view.
[0139] Step B4, based on the thermal value of each pixel point in each frame of the bird's-eye view, determine the thermal area of the wheel blocking rod in each frame of the bird's-eye view.
[0140] For each frame of the bird's-eye view, an area consisting of pixel points whose thermal values are greater than a preset value in the bird's-eye view is used as a thermal area of the wheel stopper.
[0141] Step B5, dividing the wheel blocking rod thermal area in each frame of the bird's-eye view into the wheel blocking rod connected area, and obtaining each wheel blocking rod in each frame of the bird's-eye view.
[0142] For the wheel stopper thermal area of each frame of the bird's-eye view, the thermal value of each pixel in the area is traversed to divide each pixel into foreground (wheel stopper) and background. The pixel with a thermal value greater than or equal to the threshold can be regarded as the foreground, otherwise it is the background. The global threshold or adaptive threshold method can be used to select a suitable threshold to maximize the distinction between the foreground and the background. The wheel stopper connected area is marked for the divided bird's-eye view. The connected area refers to the area where the pixels are connected. The DFS (Depth-First-SeBrch, depth-first search) or BFS (BreBdth-FirstSeBrch, breadth-first search) algorithm can be used to traverse each pixel in the divided bird's-eye view. If the pixel is the foreground (wheel stopper), its connected area is marked, and all the pixels in the connected area are marked with the same label to obtain the wheel stopper connected area. Each pixel point in the connected area of the wheel stopper corresponds to a wheel stopper weight. The specific method for determining the weight is similar to the method for determining the parking space weight in step B3 and is not repeated here. This is so that when the wheel stopper in the bird's-eye view is subsequently mapped to the target map, the wheel stopper weights corresponding to each pixel point in the bird's-eye view are mapped to the corresponding pixel point on the target map and weighted to determine the map wheel stopper in the target map.
[0143] Optionally, after obtaining the connected area of the wheel blocking rod, a contour detection algorithm (such as CBnny edge detection) can be used to extract the contour of each connected area of the wheel blocking rod, so as to facilitate the subsequent more accurate extraction and analysis of the connected area of the wheel blocking rod. A refinement algorithm (such as ZhBng-Suen algorithm) can also be used to refine the extracted contour to eliminate redundant pixels to make the connected area more accurate, and the refined connected area can be post-processed, such as removing noise, smoothing, etc., to improve the accuracy of the result.
[0144] Step B6: determine the image composed of pixels whose parking space weights are greater than or equal to the second threshold in each frame of the bird's-eye view as a parking space.
[0145] The granularity of the wheel stopper connected area division of the wheel stopper heat map is controlled at the level of the bird's-eye view map with higher accuracy and relatively more data, so that the divided wheel stopper can be mapped to the target map later, so as to quickly and accurately obtain the map wheel stopper in the target map, avoid dividing the wheel stopper in the target map with a larger field of view and prone to errors, improve the accuracy of wheel stopper recognition in the map, and avoid dividing the wheel stopper connected area for each image in the image sequence, thereby improving the efficiency of wheel stopper recognition.
[0146] Optionally, based on the coordinate mapping relationship between the image and the map, the parking spaces and wheel stoppers identified on each image in the image sequence can be mapped to a bird's-eye view and then mapped to the target map, so as to subsequently calculate the maximum actual distance to the target. Then, correspondingly, based on the inverse coordinate mapping relationship between the pre-acquired image and the map, the map parking spaces and wheel stoppers predetermined in the target map can be mapped to a bird's-eye view and then mapped to each image in the image sequence, so as to call the image in subsequent processes such as automatic parking. The specific method of mapping the map parking spaces and wheel stoppers predetermined in the target map to a bird's-eye view is introduced in the subsequent embodiments.
[0147] The coordinate inverse mapping relationship between the image and the map includes the coordinate mapping relationship between the map and the bird's-eye view and the coordinate mapping relationship between the image and the bird's-eye view. Specifically, the coordinate inverse mapping relationship between the image and the map is used to indicate any point with coordinates pcamt on an image at the tth frame (i.e., time t) in the image sequence. Mapping it to the map can obtain the corresponding coordinates pm. The specific mapping process is as follows:
[0148] (1) Map pcamt to bev of frame t: pbt = mapping1(pcamt), where mapping1 represents the coordinate mapping relationship between the image and the bird's-eye view.
[0149] (2), map pbt to the map: pm = Twgt × pbt, the entire process of (2) is used to indicate the coordinate mapping relationship between the map and the bird's-eye view.
[0150] The coordinate mapping relationship between the image and the bird's-eye view is used to indicate that any point with coordinates pb on the bird's-eye view of the tth frame is mapped to an image of the tth frame in the image sequence to obtain the corresponding coordinates pcam. It is still assumed that the coordinates of the point in the camera coordinate system of the corresponding camera of the image are pc=(x,y,z) T , the specific mapping process is as follows:
[0151] (1) Convert the points on the bird's-eye view map to the ground coordinate system: pg = (pb - n / 2) / n × lp, where pg[1] = -pg[1], n represents the number of pixels in the bird's-eye view map, lp represents the distance of each pixel in the bird's-eye view map, and pg represents the corresponding coordinates of the points on the bird's-eye view map in the ground coordinate system.
[0152] (2) Convert the point on the ground coordinate system to the coordinate of the center of the bottom of the vehicle. The mapping equation of the ground corresponding to the bottom of the vehicle is Teg: pe = Teg × pg, where pe represents the coordinate of the point on the bird's-eye view at the center of the bottom of the vehicle.
[0153] (3) Convert pe to the camera coordinate system: pc = Tce × pe, where Tce is the external parameter of the camera that captured the image, and pc represents the coordinates of the point on the bird's-eye view in the camera coordinate system.
[0154] (4) Convert pc to the pixel coordinate system of the camera: pcam=(K×pc[:3])[:2], where K is the camera intrinsic parameter, [:3] means taking the first three or two digits of pc, and [:2] means taking the first two digits of K×pc[:3]. pcam is the coordinate of the point on the bird's-eye view in the image.
[0155] Step S404, performing a nearest neighbor matching operation on each mapped parking space and the mapped wheel stopper to determine the mapped wheel stopper in each mapped parking space in the target map.
[0156] Step S405: for each mapped parking space, determine the maximum actual distance between each pixel point of the wheel stopper on the mapped parking space and the target straight line.
[0157] Step S406, for each map parking space, sort the maximum actual distances of all determined pixel points, and select the target maximum actual distance based on the sorting, the target actual maximum distance being the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold, and use the target maximum actual distance as the annotation result of the map wheel stop in the map parking space, to obtain the annotation results of each map wheel stop in the target map.
[0158] For details of steps S404 to S406, see Figure 1 Steps S104 to S106 are shown.
[0159] In the embodiment of the present application, the parking spaces and wheel stoppers identified in each image in the image sequence are first mapped to each frame of the bird's-eye view, and the wheel stopper heat map is divided into connected areas in each frame of the bird's-eye view, and the granularity of the division is controlled at the level of the bird's-eye view with higher accuracy and relatively more data, thereby improving the accuracy and efficiency of wheel stopper identification. The parking spaces and wheel stoppers in each frame of the bird's-eye view are then mapped to the target map, so as to quickly and accurately determine the map parking spaces and map wheel stoppers in the target map, and avoid the low accuracy and efficiency caused by the operation of identifying and classifying parking spaces and wheel stoppers on the entire target map, thereby quickly and accurately completing the marking of the map wheel stoppers in the target map.
[0160] In an embodiment of the present application, another wheel stopper marking method is provided, which can be used for a computer device, such as a vehicle-mounted processor, a calculator, a server, or a device with data calculation and data analysis processing capabilities. Figure 5 FIG. 1 is a flow chart of another wheel blocking rod marking method according to an embodiment of the present application. Figure 5 As shown, the process includes the following steps:
[0161] Step S501, receiving a request for marking a wheel stopper in a target bird's-eye view based on a target map, and acquiring a target bird's-eye view corresponding to the target map.
[0162] The request for marking the wheel stopper in the target bird's-eye view based on the target map includes the time corresponding to the target bird's-eye view, and the bird's-eye view corresponding to the time can be found from each frame of the bird's-eye view corresponding to the target map. If the request includes multiple time, it means there are multiple target bird's-eye views.
[0163] Step S502, based on the coordinate mapping relationship between the pre-acquired map and the bird's-eye view, map parking spaces and map wheel stoppers predetermined in the target map are mapped into the target bird's-eye view to obtain parking spaces and wheel stoppers in the target bird's-eye view.
[0164] Based on the coordinate mapping relationship between the map and the bird's-eye view, the coordinates of each pixel point on the map parking space and the map wheel stopper predetermined in the target map sheet in the target bird's-eye view are calculated, and then the annotation results of each pixel point on the map parking space and the map wheel stopper are mapped to the pixel point indicated by the coordinate, thereby obtaining the parking space and wheel stopper in the target bird's-eye view.
[0165] It should be noted that, in addition to the map parking spaces and map wheel stoppers determined based on the wheel stopper marking method introduced in any of the above embodiments, the map parking spaces and map wheel stoppers pre-marked in the target map may also be map parking spaces and map wheel stoppers pre-marked in the target map. In this way, in scenes that have not been marked using the wheel stopper marking method introduced in any of the above embodiments or in scenes where the marking effect is not good after use, pre-marking can be performed on the target map to obtain a more accurate marking result of the wheel stoppers in the bird's-eye view.
[0166] Optionally, parking spaces may be pre-marked in the scene map, and based on the coordinate mapping relationship between the map and the bird's-eye view, the parking spaces in the scene map are mapped to the target bird's-eye view, and the wheel stoppers predetermined in the target map are mapped to the target bird's-eye view.
[0167] Step S503: performing a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view.
[0168] Step S503 and Figure 4 Step S404 in the illustrated embodiment is similar, except that the nearest neighbor matching operation in the target map is changed to the nearest neighbor matching operation in the target bird's-eye view map, which will not be described in detail here.
[0169] Step S504: for each parking space in the target bird's-eye view, determine the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line.
[0170] Step S505: For each parking space in the target bird's-eye view, sort the maximum actual distances of all the determined pixel points, and select the target maximum actual distance based on the sorting. The target maximum actual distance is the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold. The target maximum actual distance is used as the labeling result of the wheel stopper in the parking space, and the labeling result of each wheel stopper in the target bird's-eye view is obtained.
[0171] Steps S504 to S505 and Figure 4 In the illustrated embodiment, steps S405 to S406 are similar and are not described in detail herein.
[0172] In one application scenario, lidar (radar) posture changes are used to measure vehicle posture changes in order to determine the coordinate mapping relationship between an image and a map.
[0173] For a sequence, that is, an image sequence, the tth moment is denoted as ft. Through the camera motion relationship, we can obtain a whole map, that is, build a scene map based on the image sequence.
[0174] By mapping the marked parking spaces on the map back to each frame of the bird's-eye view, the corresponding parking space markings in the bird's-eye view can be obtained. The specific process is similar to the process of mapping the predetermined parking spaces in the target map to the target bird's-eye view.
[0175] For each coordinate on the map, the corresponding RGB value is obtained through the coordinate mapping relationship and interpolation between the image and the map, and a weighted sum is performed (that is, the color value corresponding to each pixel in the scene map is determined according to the corresponding coordinates of each pixel in the scene map in each image, as well as the coordinates and color values of the pixel in each image, to obtain a color scene map); non-ground objects are removed according to the pre-acquired semantic map to obtain an RGB map, which is the target map.
[0176] Manual annotation is performed on the obtained RGB map to mark the wheel stopper. For the coordinates of each pixel point on the marked wheel stopper, the corresponding coordinates on the bird's-eye view are obtained through the coordinate mapping relationship between the map and the bird's-eye view, and then the distance from each point on the wheel stopper to the parking space entrance line is calculated. The length of the point whose distance length is approximately pct=5% (the increase in the distance of the previous pixel point is 5%) is calculated from small to large, which is the final distance from the wheel stopper to the entrance, that is, the maximum actual distance between each pixel point of the wheel stopper and the target straight line is determined, and the maximum actual distances are sorted in order from small to large and the target maximum actual distance is selected based on the sorting. The target actual maximum distance is the larger of the two maximum actual distances whose difference in the maximum actual distance between two adjacent pixel points is greater than or equal to the first threshold. The target maximum actual distance is determined as the annotation result of the wheel stopper.
[0177] In the embodiment of the present application, when it is necessary to mark the parking spaces and wheel stoppers in the bird's-eye view, the pre-marked map road stoppers and map parking spaces in the target map can be mapped back to the corresponding bird's-eye view based on the coordinate mapping relationship between the map and the bird's-eye view, so as to facilitate the subsequent determination of the maximum actual distance of the parking spaces and wheel stoppers in the bird's-eye view. There is no need to manually mark the parking spaces and wheel stoppers in the bird's-eye view, which can avoid the situation of human omission and mismarking and improve the accuracy of marking. In addition, if it is necessary to mark multiple bird's-eye views, the pre-marked map road stoppers and map parking spaces can be mapped to the corresponding positions in multiple bird's-eye views at one time through the target map for subsequent calculations, avoiding marking each bird's-eye view, reducing the data of repeated marking, and thus improving the marking efficiency of the bird's-eye view.
[0178] In an embodiment of the present application, another wheel stopper marking method is provided, which can be used in a computer device, such as a vehicle-mounted processor, a calculator, a server, or a device with data calculation and data analysis processing capabilities. Figure 6is a flow chart of another wheel blocking rod marking method according to an embodiment of the present application, such as Figure 6 As shown, the process includes the following steps:
[0179] Step S601, receiving a request for marking a wheel stopper in a target bird's-eye view, and obtaining a target bird's-eye view corresponding to the target map.
[0180] The request for marking the wheel stopper in the target bird's-eye view includes the time corresponding to the target bird's-eye view, and the bird's-eye view corresponding to the time can be found from the frames of the bird's-eye view corresponding to the target map. If the request includes multiple time, it means there are multiple target bird's-eye views.
[0181] Step S602, based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view, the wheel stopper and the parking space pre-identified in the multiple images corresponding to the target bird's-eye view are mapped to the target bird's-eye view to obtain the parking space and the wheel stopper in the target bird's-eye view.
[0182] For each target bird's-eye view, based on the coordinate mapping relationship between the map and the bird's-eye view, the corresponding coordinates of each pixel point on the target bird's-eye view in the multiple images that constitute the target bird's-eye view can be found, and then the wheel stopper weights and parking space weights corresponding to each pixel point pre-identified in each image are mapped to the target bird's-eye view, and the mapped wheel stopper weights and parking space weights are weighted to finally obtain the parking spaces and wheel stoppers in the target bird's-eye view.
[0183] Optionally, before step S602, it is also necessary to perform a step of identifying a parking space and a wheel stopper in each image in the image sequence, which step includes determining a plurality of wheel stopper partial images and a plurality of parking space partial images in the image sequence, each pixel point in each wheel stopper partial image corresponds to a wheel stopper weight, and each pixel point in each parking space partial image corresponds to a parking space weight. For details, see Figure 4 In the illustrated embodiment, step S402 is not described here. The above step S602 may include the following steps C1 to C7:
[0184] Step C1, according to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod partial image, convert each wheel blocking rod partial image into a heat map, and obtain multiple wheel blocking rod heat maps. Figure 4 Step B1 of the illustrated embodiment will not be described in detail here.
[0185] Step C2, based on the coordinate mapping relationship between the image and the bird's-eye view, determine the coordinates of each pixel point in the target bird's-eye view in each image corresponding to the target bird's-eye view, and obtain the image coordinate subset corresponding to each pixel point in the target bird's-eye view.
[0186] Based on the coordinate mapping relationship between the image and the bird's-eye view, the corresponding coordinates of each pixel point on the target bird's-eye view in the multiple images constituting the target bird's-eye view can be found, and then the image coordinate subset corresponding to each pixel point can be obtained.
[0187] Step C3, for each image coordinate subset, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image to which it belongs according to the coordinates of each pixel point in each image corresponding to the target bird's-eye view, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal value of each pixel point in each wheel stop rod thermal map, and obtain the parking space weight and thermal value corresponding to each image coordinate.
[0188] For each image coordinate in each subset of image coordinates, the image coordinate is found in the wheel stopper heat map of each of the multiple images constituting the target bird's-eye view, and the coordinate indicates the thermal value of the pixel point. If the image coordinate does not correspond to a certain pixel point in the wheel stopper heat map, the thermal value corresponding to the image coordinate can be calculated by interpolation, so as to obtain the thermal value corresponding to each image coordinate. The same operation is performed on the parking space partial map of each of the multiple images constituting the target bird's-eye view, and the parking space weight corresponding to each image coordinate can be obtained.
[0189] Step C4, for each image coordinate subset, the parking space weights and thermal values corresponding to each image coordinate in the image coordinate subset are respectively integrated into one parking space weight and one thermal value, and the integrated parking space weights and thermal values are respectively used as the parking space weights and thermal values of the corresponding pixel points in the target bird's-eye view of the image coordinate subset.
[0190] The parking space weight and thermal value corresponding to each image coordinate in the image coordinate subset corresponding to each pixel point in the target bird's-eye view are weighted respectively, and the weighted results are used to determine the parking space weight and thermal value corresponding to the pixel point.
[0191] Step C5, determining the thermal area of the wheel blocking rod in the target bird's-eye view based on the thermal value of each pixel point in the target bird's-eye view.
[0192] Step C6, dividing the wheel blocking rod thermal area in the target bird's-eye view into the wheel blocking rod connected area to obtain each wheel blocking rod in the target bird's-eye view.
[0193] Step C7: determine the image composed of pixels whose parking space weights in the target bird's-eye view are greater than or equal to the second threshold as the parking space.
[0194] Steps C5 to C7 and Figure 4 Steps B4 to B6 in the illustrated embodiment are similar and are not described in detail herein.
[0195] The parking spaces and wheel stoppers in the target bird's-eye view are mapped to the target bird's-eye view corresponding to the parking spaces and wheel stoppers pre-identified in each image, thereby realizing the recognition of the parking spaces and wheel stoppers in the target bird's-eye view. This eliminates the need for additional training models to recognize bird's-eye views with multiple perspectives and possibly changing object shapes, thereby improving the accuracy of the recognition of parking spaces and wheel stoppers in the bird's-eye view.
[0196] Step S603: Perform a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view.
[0197] Step S604: for each parking space in the target bird's-eye view, determine the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line.
[0198] Step S605: For each parking space in the target bird's-eye view, sort the maximum actual distances of all the determined pixel points, and select the target maximum actual distance based on the sorting. The target maximum actual distance is the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold. The target maximum actual distance is used as the labeling result of the wheel stopper in the parking space, and the labeling result of each wheel stopper in the target bird's-eye view is obtained.
[0199] In one application scenario, for each surround view camera on a vehicle, there is a corresponding semantic segmentation model Mw, which can segment objects such as the ground, parking spaces, and wheel stoppers, as well as a parking space detection model Ms.
[0200] The n images captured by the surround-view camera are sent to Mw to obtain the corresponding semantic segmentation map of the wheel stopper, that is, the local map of the wheel stopper.
[0201] For each point p on the bird's-eye view (a bird's-eye view, which is the target bird's-eye view that can be spliced together from n images), its coordinates on the n ring views are obtained respectively (based on the coordinate mapping relationship between the image and the bird's-eye view), and the data on the corresponding heat map (the heat map is the wheel-stop rod heat map predicted and output by Mw) is obtained by interpolation and weighted, and finally the semantic segmentation of the corresponding bird's-eye view is obtained (that is, the wheel-stop rod in the target bird's-eye view), and the corresponding heat map is converted into the wheel-stop rod on the bird's-eye view (that is, the thermal area of the wheel-stop rod in the target bird's-eye view is determined).
[0202] For the heat map of the wheel stopper, the connected area is divided, and each connected area represents the wheel stopper of a parking space, that is, the wheel stopper heat map in the target bird's-eye view is divided into connected areas of the wheel stopper;
[0203] The specific process of obtaining the parking spaces on the bird's-eye view through Ms is similar to that of obtaining the wheel stoppers on the bird's-eye view, except that there is no need to predict the heat map or divide the connected areas.
[0204] By performing nearest neighbor matching on each connected area of the parking space and the wheel stopper, a connected area of the parking space and the wheel stopper is obtained, that is, a nearest neighbor matching operation is performed on each parking space and the wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view;
[0205] Calculate the distance from each point in the connected area to the parking space entrance line, and calculate the length of the points whose distance lengths are sorted from small to large and are approximately pct=5%, which is the final distance from the wheel stopper to the entrance, that is, determine the maximum actual distance between each pixel point of the wheel stopper and the target straight line, and sort the maximum actual distances in order from small to large and select the target maximum actual distance based on the sorting. The target actual maximum distance is the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to the first threshold, and the target maximum actual distance is determined as the annotation result of the wheel stopper.
[0206] In the embodiment of the present application, by mapping the parking spaces and wheel stoppers pre-identified in each image corresponding to the target bird's-eye view into the target bird's-eye view, it is convenient to determine the wheel stoppers in the parking space in the target bird's-eye view, and select the target maximum actual distance from the maximum actual distances between each pixel point of the wheel stopper in the parking space and the target straight line to complete the labeling of the wheel stoppers in the target bird's-eye view, thereby realizing the labeling of each wheel stopper in the bird's-eye view, and eliminating the need to manually label the bird's-eye view, or to additionally train models to identify wheel stoppers and parking spaces in bird's-eye views with multiple perspectives and whose object shapes may change, thereby reducing the labeling cost and improving the labeling accuracy.
[0207] In the embodiments of the present application, a wheel blocking rod marking device is also provided, which is used to implement the above embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0208] The present application embodiment provides a wheel blocking rod marking device, such as Figure 7 As shown, including:
[0209] An acquisition module 710 is used to acquire a target map of the vehicle and an image sequence corresponding to the target map;
[0210] A recognition module 720, for recognizing a parking space and a wheel stopper in each image in the image sequence;
[0211] A mapping module 730 is used to map the parking spaces and wheel stoppers in each image to the target map based on the coordinate mapping relationship between the pre-acquired image and the map, so as to obtain each map parking space and map wheel stopper in the target map;
[0212] A matching module 740 is used to perform a nearest neighbor matching operation on each map parking space and the map wheel stopper to determine the map wheel stopper in each map parking space in the target map;
[0213] The first determination module 750 is used to determine, for each mapped parking space, the maximum actual distance between each pixel point of the wheel stopper in the mapped parking space and a target straight line, where the target straight line is a straight line away from the wheel stopper and perpendicular to the long side of the mapped parking space and where at least one corner point of the entrance of the mapped parking space is located;
[0214] The second determination module 760 is used to sort the maximum actual distances of all determined pixel points for each map parking space, and select a target maximum actual distance based on the sorting, the target actual maximum distance being the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and use the target maximum actual distance as the marking result of the map wheel stop in the map parking space to obtain the marking result of each map wheel stop in the target map.
[0215] In some optional embodiments, the recognition module includes a first determination unit, which is used to determine a plurality of partial images of wheel blocking rods and a plurality of partial images of parking spaces in the image sequence, and the coordinate mapping relationship between the image and the map includes a coordinate inverse mapping relationship between the map and the bird's-eye view and a coordinate inverse mapping relationship between the image and the bird's-eye view;
[0216] The mapping module includes:
[0217] A stitching unit is used to stitch the images collected at the same time in the image sequence into a bird's-eye view to obtain the bird's-eye view of each frame corresponding to the target map;
[0218] The second determining unit is used to determine the coordinates corresponding to each pixel point in the target map in each frame of the bird's-eye view based on the coordinate inverse mapping relationship between the map and the bird's-eye view, and obtain a set of bird's-eye view coordinates corresponding to each pixel point in the target map;
[0219] A third determining unit is used to determine, for each bird's-eye view coordinate set, the coordinates corresponding to each bird's-eye view coordinate in the bird's-eye view coordinate set in each image based on the coordinate inverse mapping relationship between the image and the bird's-eye view, so as to obtain an image coordinate set corresponding to the bird's-eye view coordinate set;
[0220] A first mapping unit is used to map each wheel stopper partial image and each parking space partial image to each frame of the bird's-eye view image according to the coordinates in each image coordinate set, the coordinates of each pixel point in each wheel stopper partial image, and the coordinates of each pixel point in each parking space partial image, so as to obtain the wheel stopper and the parking space in each frame of the bird's-eye view image;
[0221] The second mapping unit is used to map the wheel stopper and the parking space in each frame of the bird's-eye view to the target map according to the coordinates in each bird's-eye view coordinate set and the coordinates of each pixel point in each frame of the bird's-eye view, so as to obtain each map parking space and map wheel stopper in the target map.
[0222] In some optional embodiments, the first mapping unit includes:
[0223] A conversion subunit, for converting each wheel blocking rod local image into a thermal map according to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod local image, so as to obtain a plurality of wheel blocking rod thermal maps;
[0224] The first determination subunit is used to determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image coordinate set in the image to which it belongs, based on the coordinates in each image coordinate set, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal value of each pixel point in each wheel stop lever thermal map, so as to obtain the parking space weight and thermal value corresponding to each image coordinate;
[0225] An integration subunit is used for integrating the parking space weight and the thermal value corresponding to each image coordinate in each image coordinate set into one parking space weight and one thermal value, and using the integrated parking space weight and thermal value as the parking space weight and thermal value of the pixel point corresponding to the image coordinate in its corresponding bird's-eye view;
[0226] The second determination subunit is used to determine the thermal area of the wheel blocking rod in each frame of the bird's-eye view based on the thermal value of each pixel point in each frame of the bird's-eye view;
[0227] A division subunit is used to divide the wheel blocking rod connection area into the wheel blocking rod thermal area in each frame of the bird's-eye view, so as to obtain each wheel blocking rod in each frame of the bird's-eye view;
[0228] The third determining subunit is used to determine an image composed of pixel points whose parking space weights are greater than or equal to a second threshold in each frame of the bird's-eye view as a parking space.
[0229] In some optional embodiments, the device further comprises:
[0230] The acquisition module is further used to receive a request for marking the wheel blocking rod in the target bird's-eye view based on the target map, and acquire the target bird's-eye view corresponding to the target map;
[0231] The mapping module is further used to map the map parking spaces and the map wheel stop bars predetermined in the target map to the target bird's-eye view based on the coordinate mapping relationship between the pre-acquired map and the bird's-eye view, so as to obtain the parking spaces and the wheel stop bars in the target bird's-eye view;
[0232] A third determination module is used to perform a nearest neighbor matching operation on each parking space and wheel blocking rod in the target bird's-eye view to determine the wheel blocking rod in each parking space in the target bird's-eye view;
[0233] The first determination module is further used to determine, for each parking space in the target bird's-eye view, the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line;
[0234] The second determination module is also used to sort the maximum actual distances of all the determined pixel points for each parking space in the target bird's-eye view, and select the target maximum actual distance based on the sorting, the target maximum actual distance being the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and the target maximum actual distance is used as the labeling result of the wheel stopper in the parking space to obtain the labeling results of each wheel stopper in the target bird's-eye view.
[0235] In some optional embodiments, the device further comprises:
[0236] The acquisition module is further used to receive a request to mark the wheel stopper in the target bird's-eye view, and obtain the target bird's-eye view corresponding to the target map;
[0237] The mapping module is further used to map the wheel blocking rods and parking spaces pre-identified in the multiple images corresponding to the target bird's-eye view to the target bird's-eye view based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view, so as to obtain the parking spaces and wheel blocking rods in the target bird's-eye view;
[0238] The third determination module is further used to perform a nearest neighbor matching operation on each parking space and wheel blocking rod in the target bird's-eye view to determine the wheel blocking rod in each parking space in the target bird's-eye view;
[0239] The first determination module is further used to determine, for each parking space in the target bird's-eye view, the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line;
[0240] The second determination module is also used to sort the maximum actual distances of all the determined pixel points for each parking space in the target bird's-eye view, and select the target maximum actual distance based on the sorting, the target maximum actual distance being the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and the target maximum actual distance is used as the labeling result of the wheel stopper in the parking space to obtain the labeling results of each wheel stopper in the target bird's-eye view.
[0241] In some optional embodiments, the first determining unit is further used to determine a plurality of wheel blocking rod partial images and a plurality of parking space partial images in the image sequence, each pixel point in each wheel blocking rod partial image corresponds to a wheel blocking rod weight, and each pixel point in each parking space partial image corresponds to a parking space weight;
[0242] The mapping module includes:
[0243] A conversion unit, used for converting each wheel blocking rod local image into a thermal map according to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod local image, so as to obtain a plurality of wheel blocking rod thermal maps;
[0244] A fourth determining unit is used to determine the coordinates of each pixel point in the target bird's-eye view corresponding to each image corresponding to the target bird's-eye view based on the coordinate mapping relationship between the image and the bird's-eye view, and obtain an image coordinate subset corresponding to each pixel point in the target bird's-eye view;
[0245] A fifth determination unit is used to determine, for each image coordinate subset, the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image to which it belongs, according to the coordinates of each pixel point in each image corresponding to the target bird's-eye view, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal value of each pixel point in each wheel stopper thermal map, so as to obtain the parking space weight and thermal value corresponding to each image coordinate;
[0246] an integration unit, for each image coordinate subset, integrating the parking space weights and thermal values corresponding to each image coordinate in the image coordinate subset into one parking space weight and one thermal value, and using the integrated parking space weights and thermal values as the parking space weights and thermal values of the corresponding pixel points of the image coordinate subset in the target bird's-eye view;
[0247] A sixth determining unit, configured to determine a thermal area of the wheel blocking rod in the target bird's-eye view based on the thermal value of each pixel point in the target bird's-eye view;
[0248] A division unit is used to divide the wheel blocking rod thermal area in the target bird's-eye view into a wheel blocking rod connected area to obtain each wheel blocking rod in the target bird's-eye view;
[0249] The seventh determining unit is used to determine an image composed of pixel points whose parking space weights in the target bird's-eye view are greater than or equal to a second threshold as a parking space.
[0250] In some optional embodiments, the acquisition module includes:
[0251] An acquisition unit, used to acquire image sequences captured by each camera on the vehicle and a semantic map of the vehicle;
[0252] A construction unit, used for constructing a scene map including the ground on which the vehicle is traveling, the parking space on the ground, and the wheel blocking rod on the ground based on the image sequence;
[0253] A seventh determining unit, configured to determine the coordinates corresponding to each pixel point in the scene map in each image based on a coordinate mapping relationship between the image and the map;
[0254] an eighth determination unit, configured to determine the color value corresponding to each pixel point in the scene map according to the coordinates corresponding to each pixel point in the scene map in each image, and the coordinates and color values of the pixel point in each image, so as to obtain a color scene map;
[0255] The deletion unit is used to delete objects other than the ground, the parking spaces on the ground and the wheel blocking rods on the ground in the color scene map according to the annotation information of the scene around the vehicle in the semantic map to obtain a target map.
[0256] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0257] The wheel stop lever marking device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0258] The present application also provides a computer device having the above Figure 7 The wheel stop lever marking device shown.
[0259] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8A processor 10 is taken as an example.
[0260] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0261] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0262] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0263] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0264] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0265] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0266] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A wheel blocking rod marking method, characterized in that: The method comprises: Acquire a target map of the vehicle and an image sequence corresponding to the target map; identifying a parking space and a wheel stopper in each image in the image sequence; Based on the coordinate mapping relationship between the pre-acquired image and the map, the parking space and the wheel stopper in each of the images are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map; Performing a nearest neighbor matching operation on each of the map parking spaces and the map wheel stopper to determine a map wheel stopper in each of the map parking spaces in the target map; For each of the mapped parking spaces, determining the maximum actual distance between each pixel point of the wheel stopper in the mapped parking space and a target straight line, wherein the target straight line is a straight line away from the wheel stopper and perpendicular to the long side of the mapped parking space and where at least one corner point of the entrance of the mapped parking space is located; For each of the map parking spaces, the maximum actual distances of all the determined pixel points are sorted, and a target maximum actual distance is selected from them based on the sorting, the target actual maximum distance being the larger of two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and the target maximum actual distance is used as the marking result of the map wheel stop in the map parking space to obtain the marking results of each map wheel stop in the target map.
2. The method according to claim 1, characterized in that The identifying the parking space and the wheel blocking rod in each image in the image sequence includes: Determine a plurality of partial images of wheel blocking rods and a plurality of partial images of parking spaces in the image sequence, wherein the coordinate mapping relationship between the image and the map includes a coordinate inverse mapping relationship between the map and the bird's-eye view and a coordinate inverse mapping relationship between the image and the bird's-eye view; The method of mapping the parking space and the wheel stopper in each image to the target map based on the coordinate mapping relationship between the pre-acquired image and the map, and obtaining each map parking space and map wheel stopper in the target map, comprises: splicing the images collected at the same time in the image sequence into a bird's-eye view to obtain the bird's-eye view frames corresponding to the target map; Based on the coordinate inverse mapping relationship between the map and the bird's-eye view, the coordinates corresponding to each pixel point in the target map in each frame of the bird's-eye view are determined to obtain a set of bird's-eye view coordinates corresponding to each pixel point in the target map; For each of the bird's-eye view coordinate sets, based on the coordinate inverse mapping relationship between the image and the bird's-eye view, determine the coordinates corresponding to each of the bird's-eye view coordinates in the bird's-eye view coordinate set in each of the images, to obtain an image coordinate set corresponding to the bird's-eye view coordinate set; According to the coordinates in each of the image coordinate sets, the coordinates of each pixel point in each of the wheel stopper partial images and the coordinates of each pixel point in each of the parking space partial images, each of the wheel stopper partial images and the parking space partial images are respectively mapped to each frame of the bird's-eye view image to obtain the wheel stopper and the parking space in each frame of the bird's-eye view; According to the coordinates in each of the bird's-eye view coordinate sets and the coordinates of each pixel point in each frame of the bird's-eye view, the wheel stopper and parking space in each frame of the bird's-eye view are respectively mapped to the target map to obtain each map parking space and map wheel stopper in the target map.
3. The method according to claim 2, characterized in that Each pixel point in each of the wheel stopper partial images corresponds to a wheel stopper weight, and each pixel point in each of the parking space partial images corresponds to a parking space weight. According to the coordinates in each of the image coordinate sets, the coordinates of each pixel point in each of the wheel stopper partial images, and the coordinates of each pixel point in each of the parking space partial images, each of the wheel stopper partial images and the parking space partial images are respectively mapped to each frame of the bird's-eye view image to obtain the wheel stopper and the parking space in each frame of the bird's-eye view, including: According to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod partial image, each wheel blocking rod partial image is converted into a thermal map to obtain a plurality of wheel blocking rod thermal maps; According to the coordinates in each of the image coordinate sets, the coordinates of each pixel point in each of the parking space local maps, and the coordinates and thermal values of each pixel point in each of the wheel stop lever thermal maps, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image to which it belongs, and obtain the parking space weight and thermal value corresponding to each image coordinate; For each image coordinate in each of the image coordinate sets, the parking space weight and the thermal value corresponding to the image coordinate are respectively integrated into a parking space weight and a thermal value, and the integrated parking space weight and thermal value are respectively used as the parking space weight and thermal value of the pixel point corresponding to the image coordinate in its corresponding bird's-eye view; Based on the thermal value of each pixel point in each frame of the bird's-eye view, the thermal area of the wheel blocking rod in each frame of the bird's-eye view is determined respectively; The wheel blocking rod thermal area in each frame of the bird's-eye view is divided into the wheel blocking rod connected area to obtain each wheel blocking rod in each frame of the bird's-eye view; An image composed of pixel points whose parking space weights are greater than or equal to the second threshold in each frame of the bird's-eye view image is determined as a parking space.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: receiving a request for marking the wheel blocking lever in the target bird's-eye view based on the target map, and acquiring the target bird's-eye view corresponding to the target map; Based on the coordinate mapping relationship between the pre-acquired map and the bird's-eye view, the map parking space and the map wheel stopper predetermined in the target map are mapped into the target bird's-eye view to obtain the parking space and the wheel stopper in the target bird's-eye view; Performing a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view; For each parking space in the target bird's-eye view, determining the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line; For each parking space in the target bird's-eye view, the maximum actual distances of all the determined pixel points are sorted, and the target maximum actual distance is selected based on the sorting, the target maximum actual distance being the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold, and the target maximum actual distance is used as the labeling result of the wheel stopper in the parking space to obtain the labeling results of each wheel stopper in the target bird's-eye view.
5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Receive a request to mark the wheel stopper in the target bird's-eye view, and obtain the target bird's-eye view corresponding to the target map; Based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view, the wheel blocking rods and parking spaces pre-identified in the multiple images corresponding to the target bird's-eye view are mapped to the target bird's-eye view to obtain the parking spaces and wheel blocking rods in the target bird's-eye view; Performing a nearest neighbor matching operation on each parking space and wheel stopper in the target bird's-eye view to determine the wheel stopper in each parking space in the target bird's-eye view; For each parking space in the target bird's-eye view, determining the maximum actual distance between each pixel point of the inner wheel stopper and the target straight line; For each parking space in the target bird's-eye view, the maximum actual distances of all the determined pixel points are sorted, and the target maximum actual distance is selected based on the sorting, the target maximum actual distance being the larger of the two maximum actual distances whose difference between the maximum actual distances of two adjacent pixel points is greater than or equal to a first threshold, and the target maximum actual distance is used as the labeling result of the wheel stopper in the parking space to obtain the labeling results of each wheel stopper in the target bird's-eye view.
6. The method according to claim 5, characterized in that The identifying the parking space and the wheel blocking rod in each image in the image sequence includes: Determine a plurality of wheel stopper partial images and a plurality of parking space partial images in the image sequence, each pixel point in each of the wheel stopper partial images corresponds to a wheel stopper weight, and each pixel point in each of the parking space partial images corresponds to a parking space weight; The method of mapping the wheel blocking rods and parking spaces pre-identified in the plurality of images corresponding to the target bird's-eye view to the target bird's-eye view based on the coordinate mapping relationship between the pre-acquired image and the bird's-eye view to obtain the parking spaces and wheel blocking rods in the target bird's-eye view includes: According to the wheel blocking rod weight corresponding to each pixel point in each wheel blocking rod partial image, each wheel blocking rod partial image is converted into a thermal map to obtain a plurality of wheel blocking rod thermal maps; Based on the coordinate mapping relationship between the image and the bird's-eye view, determine the coordinates corresponding to the coordinates of each pixel point in the target bird's-eye view in each image corresponding to the target bird's-eye view, and obtain an image coordinate subset corresponding to each pixel point in the target bird's-eye view; For each of the image coordinate subsets, according to the coordinates of each pixel point in each image corresponding to the target bird's-eye view, the coordinates of each pixel point in each parking space local map, and the coordinates and thermal values of each pixel point in each wheel stopper rod thermal map, determine the parking space weight and thermal value corresponding to the pixel point indicated by each image coordinate in the image to which it belongs, and obtain the parking space weight and thermal value corresponding to each image coordinate; For each of the image coordinate subsets, the parking space weights and thermal values corresponding to the image coordinates in the image coordinate subsets are respectively integrated into one parking space weight and one thermal value, and the integrated parking space weights and thermal values are respectively used as the parking space weights and thermal values of the corresponding pixel points of the image coordinate subset in the target bird's-eye view; Determine the wheel blocking rod thermal area in the target bird's-eye view based on the thermal value of each pixel point in the target bird's-eye view; The wheel blocking rod thermal region in the target bird's-eye view is divided into a wheel blocking rod connected region to obtain each wheel blocking rod in the target bird's-eye view; An image composed of pixel points whose parking space weights in the target bird's-eye view are greater than or equal to a second threshold is determined as a parking space.
7. The method according to claim 6, characterized in that The step of acquiring a target map of the vehicle and an image sequence corresponding to the target map includes: Acquire the image sequence captured by each camera on the vehicle and a semantic map of the vehicle; constructing a scene map including the ground on which the vehicle is traveling, the parking space on the ground, and the wheel blocking rod on the ground based on the image sequence; Based on the coordinate mapping relationship between the image and the map, determining the coordinates corresponding to each pixel point in the scene map in each of the images; Determine the color value corresponding to each pixel point in the scene map according to the coordinates corresponding to each pixel point in each of the images, and the coordinates and color values of the pixel points in each of the images, to obtain a color scene map; According to the annotation information of the scene around the vehicle in the semantic map, objects other than the ground, the parking space on the ground and the wheel blocking rod on the ground in the color scene map are deleted to obtain the target map.
8. A wheel stopper rod marking device, characterized in that: The device comprises: An acquisition module, used to acquire a target map of the vehicle and an image sequence corresponding to the target map; A recognition module, used for recognizing a parking space and a wheel stopper in each image in the image sequence; A mapping module, for mapping the parking spaces and wheel stoppers in each image to the target map based on the coordinate mapping relationship between the pre-acquired image and the map, to obtain each map parking space and map wheel stopper in the target map; A matching module, used for performing a nearest neighbor matching operation on each of the map parking spaces and the map wheel stopper to determine the map wheel stopper in each of the map parking spaces in the target map; A first determination module is used to determine, for each of the mapped parking spaces, the maximum actual distance between each pixel point of the wheel stopper in the mapped parking space and a target straight line, wherein the target straight line is a straight line away from the wheel stopper and perpendicular to the long side of the mapped parking space and where at least one corner point of the entrance of the mapped parking space is located; The second determination module is used to sort the maximum actual distances of all the determined pixel points for each of the map parking spaces, and select a target maximum actual distance based on the sorting, wherein the target actual maximum distance is the larger of the two maximum actual distances whose difference in the maximum actual distances between two adjacent pixel points is greater than or equal to a first threshold, and use the target maximum actual distance as the marking result of the map wheel stop in the map parking space to obtain the marking results of each map wheel stop in the target map.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the wheel block lever marking method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are loaded and executed by a processor to implement the wheel block lever marking method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Parking stall detection method and device
CN106652551A
Indoor vehicle positioning and navigation system and method based on video image processing
CN107067794A
Automatic parking system and method
CN111169468A
Parking frame identification method and device, electronic equipment and storage medium
CN114332814A
