A vehicle mapping method, electronic equipment, vehicle and storage medium

Abstract

The application relates to a vehicle mapping method, an electronic device, a vehicle and a storage medium. The method acquires an image of a vehicle periphery; performs semantic segmentation on the image to obtain a semantic segmentation result; extracts a parking space feature based on the semantic segmentation result to obtain parking space line information and corner point information of a parking space; obtains parking space information according to preset parking space size information, the parking space line information and the corner point information; and updates a semantic map based on the parking space information. The application solves the problem of serious information loss in the mapping process and can more accurately perform vehicle mapping.

Description

Technical Field

[0001] This application relates to the technical field of vehicle mapping and positioning, and more specifically to a vehicle mapping method, electronic device, vehicle, and storage medium. Background Technology

[0002] High-precision positioning and mapping are key technologies for achieving automated parking. Currently, most autonomous driving positioning methods rely on vehicle-mounted devices such as wheel speed sensors and inertial measurement units (IMUs), or combine them with Simultaneous Localization and Mapping (SLAM). SLAM is further divided into LiDAR-based LiDAR-based LiDAR mapping and camera-based Visual LiDAR mapping. LiDAR mapping is constrained by the cost of LiDAR and is unlikely to be mass-produced in vehicle systems in the short term. Compared to LiDAR, cameras are cheaper and acquire richer image information.

[0003] In related technologies, visual map construction is mostly based on pixel-level feature points, using only corner points or edges to extract road signs. Due to the limitation of its imaging range, parking spaces on both sides of the road cannot be completely acquired, resulting in serious loss of parking space information after map construction. Summary of the Invention

[0004] This application is proposed to address the aforementioned problems. According to this application, a vehicle mapping method is provided, comprising the following steps: acquiring an image of the vehicle's surroundings; performing semantic segmentation on the image to obtain a semantic segmentation result; extracting parking space features based on the semantic segmentation result to obtain parking space line information and corner information; obtaining parking space information based on preset parking space size information, the parking space line information, and the corner information; and updating the semantic map based on the parking space information.

[0005] In one embodiment of this application, the step of extracting parking space features based on the semantic segmentation result to obtain parking space line information and corner information includes: performing straight line detection based on the semantic image of the parking space line in the semantic segmentation result to identify straight line segments of parking space corners; obtaining the line length and endpoint coordinates of the straight line segments to obtain the parking space line information and corner information, wherein the two endpoints of the straight line segments are two adjacent parking space corners; obtaining the coordinates of all parking space corners based on the parking space line information and the corner information to obtain the complete parking space information.

[0006] In one embodiment of this application, the preset parking space size information includes a preset length-to-width ratio of the parking space; obtaining parking space information based on the preset parking space size information, the parking space line information, and the corner point information includes: taking the length of the straight line segment as a first distance, the first distance being the length of the first side of the parking space; calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and the preset length-to-width ratio of the parking space as a second distance; calculating the coordinates of the remaining two corner points of the parking space based on the first distance, the second distance, and the coordinates of the two adjacent corner points of the parking space, so as to obtain the coordinates of all parking space corner points as the parking space information.

[0007] In one embodiment of this application, the step of calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and a preset length-to-width ratio of the parking space includes: comparing the first distance with a preset threshold; when the first distance is greater than the preset threshold, calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and a first preset length-to-width ratio of the parking space; when the first distance is less than or equal to the preset threshold, calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and a second preset length-to-width ratio of the parking space, wherein the first preset length-to-width ratio of the parking space is greater than the second preset length-to-width ratio of the parking space.

[0008] In one embodiment of this application, the first preset parking space length-to-width ratio is the length-to-width ratio of a parallel parking space, and the second preset parking space length-to-width ratio is the length-to-width ratio of a perpendicular parking space.

[0009] In one embodiment of this application, after updating the semantic map based on the parking space information, the method further includes: obtaining the vehicle's odometer information, calculating the vehicle's first pose in the semantic map based on the odometer information; performing loop closure detection on the image, and performing nonlinear optimization on the semantic map to obtain the vehicle's optimized second pose in the semantic map and the optimized semantic map.

[0010] In one embodiment of this application, the step of performing loop closure detection on the image and nonlinear optimization on the semantic map to obtain an optimized second pose of the vehicle in the semantic map and an optimized semantic map includes: detecting visual feature descriptors and semantic category descriptors on the image, wherein the semantic category descriptors are used to describe the semantic category information of the image; performing loop closure detection on the image based on the visual feature descriptors and the semantic category descriptors, and performing nonlinear optimization on the semantic map to obtain an optimized second pose of the vehicle in the semantic map and an optimized semantic map.

[0011] In one embodiment of this application, the image includes a current frame and historical frames. The step of performing loop closure detection on the image based on the visual feature descriptor and the semantic category descriptor, and performing nonlinear optimization on the semantic map to obtain an optimized second pose of the vehicle in the semantic map and an optimized semantic map includes: calculating a first similarity score between the visual feature descriptor of the current frame of the image and the visual feature descriptor of each frame in the historical frames; calculating a second similarity score between the semantic category descriptor of the current frame of the image and the semantic category descriptor of each frame in the historical frames; using the sum of the first similarity score and the second similarity score as a loop closure detection score; determining the optimal loop closure frame from the historical frames based on the loop closure detection score; calculating the relative pose of the vehicle between the current frame and the optimal loop closure frame; and optimizing the semantic map based on the relative pose to obtain an optimized second pose of the vehicle in the semantic map and an optimized semantic map.

[0012] In one embodiment of this application, the method further includes obtaining the vehicle's localization result based on the updated semantic map, specifically including: converting the pixels of the current frame of the image into current point cloud data; and matching the current point cloud data with the point cloud data corresponding to the updated semantic map to obtain the vehicle's pose in the semantic map.

[0013] According to another aspect of this application, an electronic device is provided, the device including a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the vehicle mapping method as described above.

[0014] According to another aspect of this application, a vehicle is provided, the vehicle being equipped with the aforementioned electronic equipment.

[0015] According to another aspect of this application, a storage medium is provided, on which a computer program is stored, which, when running, executes the vehicle mapping method described above.

[0016] The vehicle mapping method of this application performs semantic segmentation on the image around the vehicle to obtain semantic segmentation results and generate a semantic map. When generating the semantic map, features are extracted to obtain parking space line information and corner information. Then, based on preset parking space size information, parking space line information and corner information, complete parking space information is obtained for updating the semantic map. In this method, parking spaces are completed based on partial parking space line information and parking space corner information to obtain complete parking space information, which solves the problem of serious information loss during the mapping process and can perform vehicle mapping more accurately. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A flowchart of the vehicle mapping method in an embodiment of this application is shown.

[0019] Figure 2 A schematic diagram illustrating the completion of parking space information in an embodiment of this application is shown.

[0020] Figure 3 A schematic diagram of vertical and parallel parking spaces is shown in an embodiment of this application.

[0021] Figure 4 A flowchart illustrating the mapping and vehicle positioning process in an embodiment of this application is shown.

[0022] Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0024] like Figure 1 As shown in the figure, this application discloses a vehicle mapping method, which includes the following steps:

[0025] S11. Obtain images of the area surrounding the vehicle.

[0026] S12. Perform semantic segmentation on the image to obtain the semantic segmentation result.

[0027] S13. Extract parking space features based on the semantic segmentation results to obtain parking space line information and corner information.

[0028] S14. Obtain parking space information based on preset parking space size information, parking space line information, and corner point information.

[0029] S15. Update the semantic map based on the parking space information.

[0030] The vehicle mapping method of this application performs semantic segmentation on the image around the vehicle to obtain semantic segmentation results and generate a semantic map. When generating the semantic map, parking space line information and corner information are obtained by extracting features. Then, based on the preset parking space size information, parking space line information and corner information, complete parking space information is obtained for updating the semantic map. In this method, the parking space is completed by supplementing the parking space information based on some parking space line information and corner information, which solves the problem of serious information loss during the mapping process and can perform vehicle mapping more accurately.

[0031] For example, the image is a panoramic view of an AroundViewMonitor (AVM) system, hereinafter referred to as a panoramic panoramic view.

[0032] When acquiring a panoramic view, images from all four directions of the vehicle are captured and stitched together according to calibrated parameters to generate the final panoramic view.

[0033] Specifically, the images in four directions include front view, rear view, left view, and right view images captured by four cameras (front, rear, left, and right). The Zhang Zhengyou calibration method was used to correct the fisheye image and eliminate distortion during distortion coefficient calibration.

[0034] The images from the four directions mentioned above are also stitched together with the top-down view. By placing a calibration board image on the ground plane around the vehicle, the transformation matrix from the camera to the ground is calculated to transform the fisheye image to the ground coordinate system and then convert the fisheye image into a top-down view. The top-down view is then stitched together with the aforementioned images. During the stitching process, feature point matching is performed on the overlapping parts of the images to obtain corresponding points. These corresponding points are then aligned, while the non-overlapping parts of the images are retained to achieve image stitching. Finally, a panoramic surround view is obtained after stitching the images as described above.

[0035] For example, semantic segmentation of the panoramic view is achieved through a neural network model. During semantic segmentation, ground traffic markings are identified, including pedestrian crossings, arrows, speed bumps, parking lines, and other signs and background.

[0036] For example, step S13 of the above method extracts parking space features based on semantic segmentation results to obtain parking space line information and corner information, and obtains parking space information based on preset parking space size information, parking space line information, and corner information, including the following steps:

[0037] First, straight line detection is performed on the semantic image of parking space lines from the semantic segmentation results to identify straight line segments at parking space corners. Then, the length of the straight line segments and the coordinates of their endpoints are obtained to acquire parking space line information and corner point information. The two endpoints of the straight line segments are the two adjacent parking space corner points. Finally, the coordinates of all parking space corner points are obtained based on partial parking space corner information to obtain complete parking space information. The preset parking space size information includes preset parking space length-width ratios, specifically including a first preset parking space length-width ratio and a second preset parking space length-width ratio.

[0038] The process of obtaining the coordinates of all parking space corner points based on parking space line information and corner point information includes the following steps:

[0039] First, the length of the straight line segment is taken as the first distance, which is the length of the first side of the parking space. Then, based on the first distance and the preset length-to-width ratio of the parking space, the length of the second side of the parking space intersecting the straight line segment is calculated as the second distance. Finally, based on the first distance, the second distance, and the coordinates of the two adjacent corner points of the parking space, the coordinates of the remaining two corner points of the parking space are calculated to obtain the coordinates of all parking space corner points.

[0040] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating parking space information completion in an embodiment of this application. After semantic segmentation of the panoramic view, the segmentation result is obtained. The semantic image of parking space lines is extracted from the segmentation result. The cumulative probability Hough transform is used to detect straight lines in the semantic image of parking space lines, identify the straight line segments of parking space angles, and obtain the coordinates of the two endpoints of the straight line segments. Figure 2 The two endpoints are points A and B. Points A and B are two adjacent corner points of the parking space, with coordinates A(x1, y1) and B(x2, y2) respectively. The length of the straight line segment between the two endpoints A and B, as well as the coordinates of the two endpoints, constitute partial parking space corner information.

[0041] Figure 2 Points C and D are two other corner points on the parking space. Their coordinates need to be calculated to complete the parking space information. The vector S is determined based on the coordinates of endpoints A and B and the length of the straight line segment. AB The length of the straight line segment AB is the first distance L. AB The first distance is the length of the first side of the parking space. The length of the second side of the parking space that intersects the aforementioned straight segment can be calculated from the length of the first side, as follows:

[0042] Compare the first distance with a preset threshold. When the first distance is greater than the preset threshold, calculate the length of the second side of the parking space intersecting the straight line segment according to the first distance and the first preset parking space length-width ratio. When the first distance is less than or equal to the preset threshold, calculate the length of the second side of the parking space intersecting the straight line segment according to the first distance and the second preset parking space length-width ratio, where the first preset parking space length-width ratio is greater than the second preset parking space length-width ratio.

[0043] As Figure 3 shown, Figure 3 FIG. is a schematic diagram of a parallel parking space and a perpendicular parking space. The parallel parking space and the perpendicular parking space are respectively applicable to parallel parking and reverse parking into the garage. When the vehicle travels near the parking space, the short side of the perpendicular parking space is closer to the vehicle, and the long side of the parallel parking space is closer to the vehicle. In addition, the perpendicular parking space and the parallel parking space have different length-width ratios. Figure 3 In, the horizontal lines on both sides of the vehicle are lane lines. The long side of the perpendicular parking space is perpendicular to the lane line, and the long side of the parallel parking space is parallel to the lane line. The dotted lines in the figure are the image acquisition range of the camera. It can be seen from the figure that the acquisition range of the camera covers the short side of the perpendicular parking space and the long side of the parallel parking space.

[0044] As Figures 2-3 shown, calculate the length of the line segment AD through the length of the first side, that is, through the length of the straight line segment AB. It can be calculated by the following formula:

[0045] L AD = L BC = L AB / p1, L AB > T

[0046] L AD = L BC = L AB × p2, L AB < T

[0047] Wherein, the lengths of the line segment AD and the line segment BC are the same. Here, p is the parking space length-width ratio, including the first preset parking space length-width ratio and the second preset parking space length-width ratio. The first preset parking space length-width ratio is the length-width ratio of the parallel parking space, and the second preset parking space length-width ratio is the length-width ratio of the perpendicular parking space. According to the "Rules for the Planning and Design of Parking Lots" in China, the parallel parking space is 2.8m × 7.0m, and the perpendicular parking space is 2.8m × 6.0m. Therefore, the first preset length-width ratio p1 of the parallel parking space can be calculated as 2.5, and the second preset length-width ratio p2 of the perpendicular parking space is 2.143. It can be seen that the first preset length-width ratio is greater than the second preset length-width ratio.

[0048] Wherein, T is the preset threshold, and the range is 2.8m < T < 6m, which is used to judge whether L AB is the long side or the short side. When LAB When the distance is greater than the preset threshold T, the first side can be determined to be the longer side. Therefore, the length of the line segment AD intersecting the line segment is calculated based on the length of the straight line segment, the first distance, and the first preset length-to-width ratio p1 of the parking space. When L AB When the distance is less than or equal to a preset threshold T, the first side can be determined to be the shorter side. The length of the line segment AD intersecting the straight line segment is calculated based on the first distance and the second preset parking space length-to-width ratio p2. Therefore, the length of the second side can be calculated from the length of the first side.

[0049] In the above process, the first distance, the second distance, and the coordinates of the two parking space corner points A and B are obtained, which can be used to further calculate the coordinates of the other two parking space corner points.

[0050] like Figure 2 As shown, when detecting the semantic image of parking lines, vectors AA1 and BB1 can be obtained. Because the image acquisition range on the vehicle is limited, the entire parking space cannot be included during the image acquisition stage. Given that the coordinates of endpoints A and B are known, the lengths of line segments AA1 and BB1 can be obtained. These can be used to calculate the coordinates of point C (x3, y3) and point D (x4, y4), as shown in the following calculation process:

[0051] x3=L1 / L AA1 (x 11 -x1)+x1

[0052] y3=L1 / L AA1 (y 11 -y1)+y1

[0053] x4=L1 / L BB1 (x 21 -x2)+x2

[0054] y4=L1 / L BB1 (y 21 -y2)+y2

[0055] The value of L1 is related to the type of parking space. For perpendicular parking spaces, L1 = 6m; ​​for parallel parking spaces, L1 = 2.8m. Using the above method, x3, y3, x4, and y4 are obtained, thus providing the coordinates of corner points C and D of the parking space, completing the parking space information.

[0056] The vehicle mapping method of this application detects straight lines in the segmented semantic image of parking space lines and calculates the length and direction of the part of the parking space to be supplemented by combining it with preset parking space size information, thereby completing the parking space supplementation. This makes the parking space information more complete, avoids serious loss of parking space information during the mapping process, and effectively improves the accuracy of positioning and mapping accuracy.

[0057] For example, after updating the semantic map based on parking space information in step S15, this method further includes:

[0058] The process involves acquiring the vehicle's odometer information and calculating its first pose in the semantic map based on this information. Loop closure detection is then performed on the panoramic view, and nonlinear optimization is applied to the semantic map. Finally, the optimized second pose of the vehicle in the semantic map and the optimized semantic map are obtained. The vehicle odometer includes wheel speedometers and an inertial measurement unit (IMU). The vehicle pose estimated based on the odometer information is the vehicle's first pose in the semantic map. Loop closure detection and nonlinear optimization further optimize the estimated first pose and the semantic map, resulting in the optimized second pose and the optimized semantic map.

[0059] For example, the process of performing loop closure detection on a panoramic view and nonlinear optimization on a semantic map is as follows: Visual feature descriptors and semantic category descriptors are detected on the image (panoramic view), where the semantic category descriptors are used to describe the semantic category information of the image. Loop closure detection is performed on the semantic map based on the visual feature descriptors and semantic category descriptors, and nonlinear optimization is performed on the semantic map to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map.

[0060] Vehicle pose estimation based on odometer information accumulates errors over long-term localization, which can easily affect localization accuracy. Therefore, loop closure detection is needed to correct these errors and reduce them. The panoramic view includes the current frame and historical frames. Loop closure detection is implemented using visual feature descriptors (Oriented Fast and Rotated BRIEF, or ORB) and semantic category descriptors. These two descriptors are used to search through historical frames, and the historical frame with the highest matching score is identified as the closed loop frame for optimization.

[0061] The extraction of visual feature descriptors mainly includes corner point extraction using the Features from Accelerated Segment Test (FAST) algorithm. This involves utilizing an image pyramid to increase the scale invariance of FAST corner points, calculating the gray-level centroids of the FAST corner points to increase rotation invariance, and then rotating the FAST corner points in their principal directions before extracting visual feature descriptors to describe the surrounding image region. Each visual feature descriptor corresponds to a 256-bit vector.

[0062] For example, the process of performing loop closure detection on the panoramic loop view and nonlinear optimization on the semantic map using visual feature descriptors and semantic category descriptors is as follows:

[0063] Calculate the first similarity score between the visual feature descriptor of the current frame of the panoramic view and the visual feature descriptors of each frame in the historical frames. Calculate the second similarity score between the semantic category descriptor of the current frame of the panoramic view and the semantic category descriptors of each frame in the historical frames. Sum the first and second similarity scores as the loop closure detection score.

[0064] Specifically, the process of obtaining the first similarity score, the second similarity score, and the loop closure detection score in the above process is as follows: For the first similarity score, it is first necessary to calculate the ORB bag-of-words vector V of the visual feature descriptor. d Then, the first similarity score S is calculated using the L1 norm of the distance. d The calculation process is as follows:

[0065] S d =d(V dc -V dh )

[0066] Wherein, vector V dc V represents the ORB bag-of-words vector of the current frame. dh S represents the ORB bag-of-words vector of a historical frame. d This represents the first similarity score between the visual feature descriptor of the current frame and the visual feature descriptors of each frame in the history.

[0067] For the second similarity score, we need the similarity between the semantic category descriptor of the current frame and the semantic category descriptor of each frame in the history. Specifically, we define the semantic category descriptor V. l And i = f i m i , where f i Indicates whether the i-th category appears in the current image, m i Let m be a weighted vector. i The definition is as follows:

[0068] m i =(n iIi / n Ii )×log(N / N i )

[0069] Where n Ii n represents the number of categories appearing in the current image. iIi This indicates whether the i-th category appears, and N represents the total number of images in the database. N / N i This represents the number of images in the database that include the i-th class. Based on this, the second similarity score based on semantic category can be calculated using the L1 norm of the distance, as shown in the following formula:

[0070]

[0071] Wherein, vector V lc The semantic category descriptor of the current frame, vector V lh The semantic category descriptor representing the historical frame, and the calculated S l This represents the second similarity between the semantic category descriptor of the current frame and the semantic category descriptors of each frame in the history.

[0072] After obtaining the first and second similarity scores, the loop closure detection score is obtained by summing the two scores, as shown below:

[0073] S = S d +S l

[0074] After obtaining the loop closure detection score, the optimal loop closure frame is determined from the historical frames based on the loop closure detection score. The relative pose of the vehicle between the current frame and the optimal loop closure frame is calculated. The semantic map is then nonlinearly optimized based on the relative pose to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map. Specifically, the historical frame corresponding to the highest loop closure score is taken as the optimal loop closure frame, and loop closure detection is performed on the vehicle's trajectory based on the optimal loop closure frame to obtain the loop closure detection result. The relative pose of the vehicle between the current frame and the optimal loop closure frame is calculated based on the loop closure detection result. This relative pose is used as a global constraint and, together with the pose constraints of consecutive frames, is used for semantic map optimization to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map.

[0075] For example, the panoramic view includes the current frame and historical frames. This method also includes obtaining the vehicle's localization result based on the updated semantic map, as follows:

[0076] The pixels of the current frame of the panoramic view are converted into current point cloud data. This current point cloud data is then matched with the corresponding point cloud data in the updated semantic map to obtain the vehicle's pose in the semantic map. The vehicle's pose in the semantic map obtained through this matching process is the desired vehicle localization result.

[0077] Specifically, during vehicle localization, the Iterative Nearest Point (ICP) algorithm, which integrates semantic information, is used to match the current point cloud data with the corresponding point cloud data in the updated semantic map. The matching result is then combined with the vehicle's pose in the semantic map to obtain its pose in the world coordinate system. Simultaneously, the vehicle's odometer constructs motion observation equations by measuring the vehicle's current acceleration, angular velocity, wheel speed, and other state information. This process estimates the vehicle's motion state, and combined with the vehicle's pose in the world coordinate system obtained from the above steps, vehicle localization is achieved.

[0078] like Figure 4 As shown, the method in this embodiment can be implemented through a mapping module and a positioning module in the vehicle. The vehicle positioning process can be summarized as follows:

[0079] The mapping module is used to: acquire images from four directions to generate an AVM panoramic view; perform semantic segmentation on the AVM panoramic view and identify ground traffic markings; extract parking space information and complete the parking spaces based on preset parking space size information; calculate the vehicle pose in the semantic map using vehicle odometers (IMU, wheel speedometer); and optimize the semantic map by combining semantic category information and visual feature descriptors for loop closure detection.

[0080] The localization module is used to: search for the initial position of the current vehicle in the global map and perform global initialization; and to complete the real-time localization of the vehicle based on the semantic map and in conjunction with the odometer. The initial position of the current vehicle in the global map is the vehicle's initial pose.

[0081] The vehicle mapping method of this application performs semantic segmentation on the panoramic view surrounding the vehicle to obtain semantic segmentation results and generate a semantic map. During semantic map generation, features are extracted to obtain parking space line information and corner information, which are then combined with preset parking space size information to obtain complete parking space information, and this information is used to update the semantic map. Finally, vehicle localization is performed based on the panoramic view and the semantic map. This method completes the parking space information by supplementing it with partial parking space corner information, solving the problem of severe parking space information loss after mapping in related technologies, and improving mapping efficiency and accuracy. Furthermore, this method performs loop closure detection on the panoramic view using visual feature descriptors combined with semantic category descriptors, and then performs nonlinear optimization on the semantic map, reducing errors and improving the accuracy of mapping and localization. In addition, this method matches the current point cloud data with the point cloud data corresponding to the updated semantic map, resulting in faster localization speed, higher localization accuracy, and stronger robustness.

[0082] For example, in step S12, the panoramic view image is segmented using a neural network model, which has been trained before use. The specific process is as follows:

[0083] The original image of the panoramic view and the corresponding labeled segmentation mask image are obtained. The labels on the corresponding segmentation mask image are obtained manually. These labels are used for subsequent deep learning and training to tell the computer that the places with these features are parking lines.

[0084] Training, validation, and test sets are constructed for the panoramic view. In practical applications, the training, validation, and test sets can be randomly allocated according to a certain ratio. The partitioned training set includes the segmentation mask and road sign image data after grayscale normalization and data augmentation processing.

[0085] Construct a neural network model, which is a UNet++ convolutional neural network model.

[0086] The constructed neural network is trained using labeled images as the training set to generate a network model, and the model's performance is evaluated using test set image data. Finally, the trained neural network model is used to segment the panoramic view under test and obtain the segmentation results.

[0087] like Figure 5 As shown, this application embodiment also provides an electronic device 500, including a memory 501 and a processor 502. The memory 501 stores a computer program that is run by the processor 502. When the computer program is run by the processor 502, the processor 502 executes the above-described vehicle mapping method.

[0088] This application embodiment also provides a storage medium storing a computer program, which executes the above-described vehicle mapping method when running.

[0089] This application also provides a vehicle equipped with the aforementioned electronic device, and the aforementioned vehicle mapping method is executed based on the electronic device.

[0090] The vehicle electronic device, vehicle, and storage medium in this embodiment, based on the aforementioned vehicle mapping method, perform semantic segmentation on the panoramic view surrounding the vehicle to obtain semantic segmentation results and generate a semantic map. During semantic map generation, features are extracted to obtain parking space line information and corner information, which are then combined with preset parking space size information to obtain complete parking space angle information, and this information is used to update the semantic map. Finally, vehicle localization is performed based on the panoramic view and the semantic map. This method completes the parking space angle information by supplementing it with partial parking space angle information, solving the problem of severe parking space information loss after mapping in related technologies, and improving mapping efficiency and accuracy. Furthermore, this method performs loop closure detection on the panoramic view using visual feature descriptors combined with semantic category descriptors, and performs nonlinear optimization on the semantic map, reducing errors and improving the accuracy of mapping and localization. In addition, this method matches the current point cloud data with the point cloud data corresponding to the updated semantic map, resulting in faster localization speed, higher localization accuracy, and stronger robustness.

[0091] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0094] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0095] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0096] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0097] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0098] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0099] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0100] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A vehicle mapping method, characterized in that, The method includes the following steps: Acquire images of the area surrounding the vehicle; The image is semantically segmented to obtain the semantic segmentation result; Based on the semantic segmentation results, parking space features are extracted to obtain parking space line information and corner information, including: performing straight line detection based on the semantic image of parking space lines in the semantic segmentation results to identify straight line segments of parking space corners; obtaining the line length and endpoint coordinates of the straight line segments to obtain the parking space line information and corner information, wherein the two endpoints of the straight line segments are two adjacent parking space corners; The parking space information is obtained based on the preset parking space size information, the parking space line information, and the corner point information, including: the preset parking space size information includes a preset length-to-width ratio of the parking space; the length of the straight line segment is used as a first distance, where the first distance is the length of the first side of the parking space; the length of the second side of the parking space intersecting the straight line segment is calculated based on the first distance and the preset length-to-width ratio of the parking space as a second distance; the coordinates of the remaining two corner points of the parking space are calculated based on the first distance, the second distance, and the coordinates of the two adjacent corner points of the parking space, so as to obtain the coordinates of all corner points of the parking space as the parking space information; The calculation of the length of the second side of the parking space intersecting the straight line segment based on the first distance and a preset parking space length-to-width ratio includes: comparing the first distance with a preset threshold; when the first distance is greater than the preset threshold, calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and a first preset parking space length-to-width ratio; when the first distance is less than or equal to the preset threshold, calculating the length of the second side of the parking space intersecting the straight line segment based on the first distance and a second preset parking space length-to-width ratio, wherein the first preset parking space length-to-width ratio is greater than the second preset parking space length-to-width ratio; and updating the semantic map based on the parking space information.

2. The method according to claim 1, characterized in that, The first preset parking space has the same length-to-width ratio as a parallel parking space, and the second preset parking space has the same length-to-width ratio as a perpendicular parking space.

3. The method according to any one of claims 1-2, characterized in that, After updating the semantic map based on the parking space information, the method further includes: Obtain the odometer information of the vehicle, and calculate the first pose of the vehicle in the semantic map based on the odometer information; Loop closure detection is performed on the image, and nonlinear optimization is performed on the semantic map to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map.

4. The method according to claim 3, characterized in that, The steps of performing loop closure detection on the image and nonlinear optimization on the semantic map to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map include: Visual feature descriptors and semantic category descriptors are detected on the image, wherein the semantic category descriptors are used to describe the semantic category information of the image; Based on the visual feature descriptor and the semantic category descriptor, loop closure detection is performed on the image, and nonlinear optimization is performed on the semantic map to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map.

5. The method according to claim 4, characterized in that, The image includes the current frame and historical frames. Loop closure detection is performed on the image based on the visual feature descriptor and the semantic category descriptor. Nonlinear optimization is then performed on the semantic map to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map, including: Calculate the first similarity score between the visual feature descriptor of the current frame of the image and the visual feature descriptor of each frame in the history frames; Calculate the second similarity score between the semantic category descriptor of the current frame of the image and the semantic category descriptor of each frame in the history frames; The sum of the first similarity score and the second similarity score is used as the loop closure detection score; The optimal loop closure frame is determined from the historical frames based on the loop closure detection score. Calculate the relative pose of the vehicle between the current frame and the optimal closed-loop frame; The semantic map is optimized based on the relative pose to obtain the optimized second pose of the vehicle in the semantic map and the optimized semantic map.

6. The method according to any one of claims 1-2, characterized in that, The method further includes obtaining the vehicle's location result based on the updated semantic map, specifically including: Convert the pixels of the current frame of the image into current point cloud data; The current point cloud data is matched with the point cloud data corresponding to the updated semantic map to obtain the vehicle's pose in the semantic map.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the vehicle mapping method as described in any one of claims 1-6.

8. A vehicle, characterized in that, The vehicle is equipped with the electronic equipment as described in claim 7.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, performs the vehicle mapping method as described in any one of claims 1-6.

Images