High-definition map model training method, map generation method, equipment and medium
By using road reconstruction maps for BEV feature layer prediction and auxiliary training during high-definition map model training, the problems of accuracy and information loss in high-definition map models are solved, and the accuracy of map generation and the richness of semantic information are improved.
Patent Information
- Application Number
- CN202410992116.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
How to ensure the accuracy of high-definition map models, especially to avoid information loss during the decoding process.
During the training of the high-definition map model, the road reconstruction map with color and/or coordinate information is input into the BEV feature layer for prediction. The predicted values and the true values are used for auxiliary training to increase training data and improve model accuracy. Furthermore, the predicted values output by the BEV feature layer are used directly for auxiliary training to avoid information loss during the decoding stage.
It improves the accuracy of high-definition map model-generated maps, enriches the semantic information in the decoding stage, enhances the model's training data, and avoids information loss in the decoding stage.
Smart Images

Figure CN118941729A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of high-definition map technology, and in particular to a training method, a map generation method, a device and a medium for a high-definition map model. Background Art
[0002] HD maps are a collection of vectorized static map elements, including lane lines, crosswalks, road boundaries, road centerlines, intersections, diversion areas, no-parking zones, etc. HD map models can be trained using a large amount of manually annotated or automatically generated HD map data, and then have the ability to perceive the surrounding map elements in real time relying only on visual sensor data.
[0003] In related technologies, high-definition map models usually first encode the perception data of each perspective to obtain a feature map from the BEV perspective, and then decode the BEV feature map to directly obtain a vectorized map element instance, which also avoids the heavy post-processing work in the segmentation scheme.
[0004] How to ensure the accuracy of high-definition map models is a problem that needs to be solved. Summary of the invention
[0005] In order to solve the above problems, the training method, map generation method, device and medium of the high-definition map model provided in this application can improve the accuracy of subsequent map generation by the high-definition map model.
[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a training method for a high-definition map model, wherein the high-definition map model includes a coding layer, a BEV feature layer and a decoding layer, and the method includes: inputting a road reconstruction map into the BEV feature layer to obtain a first prediction value output by the BEV feature layer; wherein the road reconstruction map has a corresponding first true value; the first true value includes color and / or coordinates; and using the first prediction value and the first true value to complete auxiliary training of the high-definition map model.
[0007] The road reconstruction image is obtained in the following manner: gridding a sequence of multiple consecutive frames of surround view images from different perspectives to obtain a first grid image; optimizing the coordinates and color of each grid point in the first grid image to obtain a second grid image; using the second grid image as the road reconstruction image, and using the coordinates and color corresponding to the second grid image as the first true value.
[0008] The method of optimizing the coordinates and color of each grid point in the first grid image to obtain the second grid image includes: performing image segmentation on the surround image sequence using a segmentation network to obtain a first image segmentation result; rendering the first grid image to obtain a two-dimensional image; performing image segmentation on the two-dimensional image using a segmentation network to obtain a second image segmentation result; obtaining a semantic segmentation loss using the first image segmentation result and the second image segmentation result; obtaining a color loss using the color in the two-dimensional image and the color in the surround image sequence; and optimizing the coordinates and color of each grid point in the first grid image according to the semantic segmentation loss and the color loss to obtain the second grid image.
[0009] Among them, according to the semantic segmentation loss and the color loss, the coordinates and the color of each grid point in the first grid image are optimized to obtain the second grid image, including: according to the semantic segmentation loss and the color loss, the internal parameters and the external parameters of the image acquisition device for acquiring the surround view image sequence are optimized; and the coordinates and the color of each grid point in the first grid image are adjusted by using the optimized internal parameters and external parameters to obtain the second grid image.
[0010] In which, in response to the semantic segmentation loss and the color loss meeting the preset conditions, the optimization of the coordinates and the color of each grid point in the first grid image is stopped.
[0011] Before the road reconstruction map is input into the BEV feature layer, the method includes: determining a rectangular area from the road reconstruction map according to the position of the vehicle, and using information corresponding to the rectangular area as the first true value.
[0012] Among them, the first predicted value and the first true value are used to complete the auxiliary training of the high-definition map model, including: using the first predicted value and the first true value to obtain a first loss value; using a feature extraction network to perform feature extraction on the first predicted value to obtain a first feature output by each layer in the feature extraction network; using the feature extraction network to perform feature extraction on the first true value to obtain a second feature output by each layer in the feature extraction network; using the first feature and the second feature output by each layer to obtain a second loss value; using the first loss value and the second loss value to complete the auxiliary training of the high-definition map model.
[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide a map generation method, the method comprising: obtaining an image and / or a point cloud; inputting the image and / or the point cloud into a high-definition map model to obtain a map output by the high-definition map model; wherein the high-definition map model is trained using the method provided in the above technical solution.
[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory coupled to the processor; wherein the memory is used to store computer programs, and the processor is used to execute the computer programs to implement a method provided by any of the above technical solutions.
[0015] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by any of the above technical solutions.
[0016] The beneficial effects of the embodiments of the present application are as follows: different from the prior art, the training method, map generation method, device and medium of the high-definition map model provided by the present application, during the training process of the high-definition map model, a road reconstruction map with a first real value such as color and / or coordinate information is input into the BEV feature layer in the high-definition map model for prediction, and a first prediction value output by the BEV feature layer is obtained, and then the first prediction value and the first real value are used to complete the auxiliary training of the high-definition map model, thereby increasing the training data of the high-definition map model and improving the accuracy of the subsequent map generation by the high-definition map model. Furthermore, the road reconstruction map has corresponding color and / or coordinate information. During the training process, the high-definition map model can learn information such as color and texture information, thereby improving the accuracy of the subsequent map generation by the high-definition map model. Furthermore, the prediction value output by the BEV feature layer is directly used for auxiliary training, thereby avoiding information loss in the decoding stage and supplementing the high-definition map model with rich semantic information in the post-decoding stage. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0018] Figure 1 It is a flowchart of an embodiment of a training method for a high-definition map model provided by the present application;
[0019] Figure 2 It is a structural schematic diagram of an embodiment of a high-definition map model provided by this application;
[0020] Figure 3 This application provides access to road reconstruction Figure 1 Schematic diagram of the process of the embodiment;
[0021] Figure 4 This application provides Figure 3A schematic diagram of a flow chart of an embodiment of step 22;
[0022] Figure 5 This application provides Figure 4 A schematic diagram of a flow chart of an embodiment of step 225;
[0023] Figure 6 It is a flowchart of another embodiment of the training method of the high-definition map model provided by the present application;
[0024] Figure 7 This application provides Figure 6 A schematic diagram of a flow chart of an embodiment of step 53;
[0025] Figure 8 It is a flowchart of an embodiment of a map generation method provided by the present application;
[0026] Fig. 9 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;
[0027] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0030] Definitions of terms used in this application:
[0031] Road reconstruction image: refers to a road image obtained by offline road reconstruction using continuous multi-frame, multi-view surround images, wherein the road reconstruction image has color and / or coordinate information.
[0032] HD maps are a collection of vectorized static map elements, including lane lines, crosswalks, road boundaries, road centerlines, intersections, diversion areas, no-parking zones, etc. HD map models can be trained using a large amount of manually annotated or automatically generated HD map data, and then have the ability to perceive the surrounding map elements in real time relying only on visual sensor data.
[0033] In related technologies, high-definition map models usually first encode the perception data of each perspective to obtain a feature map from the BEV perspective, and then decode the BEV feature map to directly obtain a vectorized map element instance, which also avoids the heavy post-processing work in the segmentation scheme.
[0034] How to ensure the accuracy of high-definition map models is a problem that needs to be solved.
[0035] Based on this, the present application proposes that during the training process of the HD map model, a road reconstruction map with first true values such as color and / or coordinate information is input into the BEV feature layer in the HD map model for prediction, and a first prediction value output by the BEV feature layer is obtained, and then the first prediction value and the first true value are used to complete the auxiliary training of the HD map model, thereby increasing the training data of the HD map model and improving the accuracy of the subsequent map generation by the HD map model. Furthermore, the road reconstruction map has corresponding color and / or coordinate information. During the training process, the HD map model can learn information such as color and texture information, and improve the accuracy of the subsequent map generation by the HD map model. Furthermore, the prediction value output by the BEV feature layer is directly used for auxiliary training, avoiding information loss in the decoding stage, and supplementing the HD map model with rich semantic information in the post-decoding stage. Please refer to any of the following embodiments for details.
[0036] See also Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a training method for a high-definition map model provided by the present application. The high-definition map model includes a coding layer, a BEV (Bird's Eye View) feature layer and a decoding layer. The method includes:
[0037] Step 11: Input the road reconstruction map into the BEV feature layer to obtain a first predicted value output by the BEV feature layer; wherein the road reconstruction map has a corresponding first true value.
[0038] The first real value includes color and / or coordinates.
[0039] In some embodiments, the coding layer of the high-definition map model can be set according to actual needs. For example, if the input of the high-definition map model includes an image, the coding layer can be set to a coding network corresponding to the image. If the input of the high-definition map model includes a point cloud, the coding layer can be set to a coding network corresponding to the point cloud. If the input of the high-definition map model includes an image and a point cloud, the coding layer can be set to two coding networks, one coding network corresponding to the image, and the other coding network corresponding to the point cloud. Further, the overall reasoning process of the high-definition map model is: the image and / or point cloud is input to the coding layer for encoding to obtain coding features, and then the coding features are converted into bird's-eye features, and then the bird's-eye features are input to the BEV feature layer, and the BEV feature layer outputs the corresponding BEV features to the decoding layer, which is decoded by the decoding layer to obtain the final prediction value. The predicted value can be used directly to generate a map.
[0040] In some embodiments, in combination Figure 2 To explain:
[0041] The image and / or point cloud is input to the encoding layer for encoding to obtain the encoding features, and the encoding features are converted into bird's-eye view features for output in the encoding layer. The output bird's-eye view features are input to the BEV feature layer, which extracts spatial features from the bird's-eye view features and outputs the corresponding BEV features. The BEV features are then input to the decoding layer, which decodes the BEV features to obtain the final prediction value.
[0042] Based on this, the high-definition map model needs to be trained before use so that the high-definition map model has good accuracy. To this end, the present application proposes to use the road reconstruction map to assist in training the high-definition map model.
[0043] In some embodiments, the type of the road reconstruction map is a bird's-eye view map, and thus can be directly input into the BEV feature layer for prediction.
[0044] Step 12: Use the first predicted value and the first true value to complete the auxiliary training of the high-definition map model.
[0045] In some embodiments, the first predicted value and the first true value may be used to calculate a loss value, and then the loss value may be used to complete auxiliary training of the high-definition map model.
[0046] In some embodiments, the number of training times of the HD map model can be adjusted according to the difference between the first predicted value and the first true value, thereby adjusting the network parameters of the HD map model. The difference between the first predicted value and the first true value can be represented by a loss value between the first predicted value and the first true value. The larger the loss value, the larger the difference between the first predicted value and the first true value.
[0047] Since the prediction accuracy of the HD map model is essentially related to the number of training times, after each training, the network parameters of the HD map model can be adjusted by gradient descent and other methods. Therefore, when the loss value is large, the number of training times is increased, and the number of times the network parameters of the HD map model are adjusted can be increased, so that the network parameters of the HD map model are adjusted to the required state.
[0048] When the loss value is small, it means that the HD map model is close to the ideal state, so the remaining training times are reduced and the training time of the HD map model is reduced.
[0049] In some embodiments, network parameters of the HD map model may be adjusted according to the difference between the first predicted value and the first true value, such as the encoding layer, decoding layer, and related parameters in the BEV feature layer in the HD map model. For example, the BEV query (bev_query), BEV pose (bev_pose), reference point (ref_point), and value (value) in the BEV feature layer. The weight matrix, activation value, and number of neurons in the encoding layer and decoding layer are connected in a certain way.
[0050] In some embodiments, in addition to using the road reconstruction map to assist in training, the HD map model also uses corresponding training samples for training, such as inputting the training samples into the encoding layer for encoding to obtain encoding features, then converting the encoding features into bird's-eye features, and then inputting the bird's-eye features into the BEV feature layer, which outputs the corresponding features to the decoding layer, which decodes them to obtain the final predicted value. The loss value is calculated based on the predicted value and the true value in the training sample, and then the HD map model is adjusted. The training samples can be composed of several images and / or point clouds, which are annotated with corresponding true values.
[0051] In some embodiments, the training sample is input to the encoding layer for encoding, and then the encoded features are converted into bird's-eye view features, and then the bird's-eye view features are input to the BEV feature layer, and the BEV feature layer outputs the corresponding features to the decoding layer, and the decoding layer decodes them to obtain the final prediction value. The loss value is calculated based on the predicted value and the true value in the training sample, and the method of adjusting the high-definition map model is defined as the main training process, and the process of using the road reconstruction map to assist training is defined as the auxiliary training process.
[0052] In some embodiments, the HD map model is comprehensively adjusted using the loss value determined by the auxiliary training process and the loss value determined by the main training process.
[0053] In some embodiments, the high-definition map model trained by the present application can be vectorized and mapped online.
[0054] In this embodiment, during the training process of the high-definition map model, a road reconstruction map having first real values such as color and / or coordinate information is input into the BEV feature layer in the high-definition map model for prediction, and a first prediction value output by the BEV feature layer is obtained. The first prediction value and the first real value are then used to complete auxiliary training of the high-definition map model, thereby increasing the training data of the high-definition map model and improving the accuracy of subsequent map generation by the high-definition map model. Furthermore, the road reconstruction map has corresponding color and / or coordinate information. During the training process, the high-definition map model can learn information such as color and texture information, thereby improving the accuracy of subsequent map generation by the high-definition map model. Furthermore, the BEV feature layer is directly used to predict the road reconstruction map, and then the prediction value is output for auxiliary training, which is equivalent to skipping the complex encoding stage, avoiding information loss in the decoding stage, and supplementing the post-decoding stage of the high-definition map model with rich semantic information.
[0055] See also Figure 3 , the road reconstruction map is obtained in the following way:
[0056] Step 21: gridding a sequence of multiple consecutive frames of surround view images at different viewing angles to obtain a first grid image.
[0057] In some embodiments, ORB-SLAM2 or COLMAP or other technologies may be used to grid a plurality of consecutive frames of surround image sequences with different viewing angles to obtain a first grid image, wherein each grid point in the first grid image has corresponding coordinates and colors.
[0058] In some embodiments, during the gridding process, the internal and external parameters of the image acquisition device that acquires the surround image sequence are obtained. The surround image sequence of multiple consecutive frames with different viewing angles is gridded in combination with the internal and external parameters to obtain a first grid image. That is, each grid point in the first grid image can be determined to have corresponding coordinates and colors in combination with the internal and external parameters.
[0059] Further, the image type of the first grid image is a bird's-eye view.
[0060] Step 22: Optimize the coordinates and color of each grid point in the first grid image to obtain a second grid image.
[0061] In some embodiments, the gridding process may have errors, and therefore needs to be optimized to obtain a second grid image that more accurately corresponds to the surround image sequence.
[0062] In some embodiments, see Figure 4 , step 22 may be the following process:
[0063] Step 221: Use a segmentation network to perform image segmentation on the surround view image sequence to obtain a first image segmentation result.
[0064] The segmentation network is a trained segmentation network. Based on this, the segmentation network is used to perform image segmentation on the surround image sequence to obtain a first image segmentation result. The first image segmentation result is used as an optimization condition for subsequent optimization.
[0065] Step 222: Render the first grid image to obtain a two-dimensional image.
[0066] In some embodiments, the first grid image may be rendered using a corresponding renderer to obtain a two-dimensional image.
[0067] In some embodiments, the first grid image may be rendered using a differentiable renderer to obtain a two-dimensional image. For example, the differentiable renderer may be PyTorch3D.
[0068] Step 223: Use the segmentation network to perform image segmentation on the two-dimensional image to obtain a second image segmentation result.
[0069] Similarly, the segmentation network is used to perform image segmentation on the two-dimensional image to obtain a second image segmentation result.
[0070] Step 224: Obtain semantic segmentation loss using the first image segmentation result and the second image segmentation result.
[0071] Since the first image segmentation result and the second image segmentation result are obtained by performing image segmentation using the same segmentation network, the first image segmentation result and the second image segmentation result can be used to obtain the semantic segmentation loss.
[0072] In some embodiments, the semantic segmentation loss is calculated using the following formula:
[0073]
[0074] Among them, L sem represents semantic segmentation loss, Mask j It is used to remove the area in the image that does not participate in the calculation. N represents the number of surround images, S j represents the second image segmentation result, represents the first image segmentation result, and CE(*) represents the cross entropy loss function.
[0075] That is, in this process, the first image segmentation result is used as the true value.
[0076] Step 225: Obtain color loss using the colors in the two-dimensional image and the colors in the surround image sequence.
[0077] Since the two-dimensional image is obtained by the surround image sequence, the color loss can be obtained by using the colors in the two-dimensional image and the colors in the surround image sequence.
[0078] For example, for each point in the first grid image, the color of the corresponding point in the surround image sequence is found according to the projection relationship, and the color of the corresponding point in the rendered two-dimensional image is found according to the renderer coordinate correspondence relationship. Then, the color loss is calculated using the colors of the corresponding points in the two-dimensional image and the surround image sequence.
[0079] In some embodiments, the color loss is calculated using the following formula:
[0080]
[0081] Among them, L color represents color loss, N represents the number of surround images, Mask j Used to remove the area in the image that does not participate in the calculation, C j Represents the color in a two-dimensional image, Represents the color in the surround image sequence. That is, in this process, the color in the surround image sequence is used as the true value.
[0082] Step 226: According to the semantic segmentation loss and the color loss, the coordinates and the color of each grid point in the first grid image are optimized to obtain a second grid image.
[0083] In some embodiments, see Figure 5 , step 225 may be the following process:
[0084] Step 41: Optimize the intrinsic parameters and extrinsic parameters of the image acquisition device for acquiring the surround view image sequence according to the semantic segmentation loss and the color loss.
[0085] In some embodiments, based on semantic segmentation loss and color loss, a gradient descent method is used to optimize the intrinsic and extrinsic parameters of an image acquisition device that acquires a surround view image sequence.
[0086] In some embodiments, based on semantic segmentation loss and color loss, Newton's method is used to optimize the intrinsic and extrinsic parameters of an image acquisition device that acquires a surround view image sequence.
[0087] Step 42: Use the optimized internal and external parameters to adjust the coordinates and color of each grid point in the first grid image to obtain a second grid image.
[0088] After the internal parameters and external parameters are optimized, the optimized internal parameters and external parameters can be used to adjust the coordinates and color of each grid point in the first grid image to obtain the second grid image.
[0089] In some embodiments, the optimization process is completed by iteration, that is, after obtaining the second grid image, the second grid image is used as the first grid image, and step 222 is performed again to obtain the corresponding semantic segmentation loss and color loss. In response to the semantic segmentation loss and color loss meeting the preset conditions, the coordinates and colors of each grid point in the first grid image are stopped from being optimized. The corresponding second grid image at this time is used as the road reconstruction map, and the coordinates and colors corresponding to the second grid image are used as the first true value. In other words, after obtaining the second grid image this time, the second grid image needs to be rendered as the first grid image to obtain a new two-dimensional image. And the new two-dimensional image is segmented to obtain a new second image segmentation result. The semantic segmentation loss is obtained using the first image segmentation result and the new second image segmentation result. The color loss is obtained using the color in the new two-dimensional image and the color in the surround image sequence. Then, according to the semantic segmentation loss and the color loss, it is determined whether to optimize the internal and external parameters of the image acquisition device for collecting the surround image sequence. If the semantic segmentation loss and the color loss meet the preset conditions, the internal and external parameters of the image acquisition device for collecting the surround image sequence are no longer optimized. The second grid image is used as the road reconstruction map. If the semantic segmentation loss and color loss do not meet the preset conditions, the internal and external parameters of the image acquisition device for acquiring the surround image sequence are optimized again to determine a new second grid image. The above process is iterated again until the semantic segmentation loss and color loss meet the preset conditions and the iteration ends.
[0090] In some embodiments, the preset condition may be that the sum of the color loss and the semantic segmentation loss is less than a threshold.
[0091] Step 23: Use the second grid image as the road reconstruction image, and the coordinates and colors corresponding to the second grid image as the first truth.
[0092] In this process, the true value of the road reconstruction map is automatically obtained by using the corresponding segmentation network, rendering, etc., without the need for manual true value labeling, thereby improving labeling efficiency.
[0093] See also Figure 6 , Figure 6 1 is a flow chart of another embodiment of the training method of the high-definition map model provided by the present application. The high-definition map model includes a coding layer, a BEV feature layer and a decoding layer. The method includes:
[0094] Step 51: Determine a rectangular area from the road reconstruction image according to the position of the vehicle, and use information corresponding to the rectangular area as the first true value.
[0095] In some embodiments, since the road reconstruction image is obtained using a surround view image sequence, the surround view image sequence substantially corresponds to the corresponding vehicle or autonomous mobile device. In this case, a rectangular area can be determined from the road reconstruction image according to the position of the vehicle, and the information corresponding to the rectangular area is used as the first true value.
[0096] For example, select a rectangular area with a range of 15m to the left and right of the vehicle and 30m in front and behind.
[0097] Step 52: Input the road reconstruction map into the BEV feature layer to obtain a first prediction value output by the BEV feature layer.
[0098] In some embodiments, a detection head is set in the BEV feature layer, and the detection head is used to output a first prediction value. The size corresponding to the first prediction value is also a prediction image of (H, W, 3). The detection head is not limited and can be composed of a combination of structures such as 2D convolution and resblock.
[0099] Step 53: Use the first predicted value and the first true value to complete auxiliary training of the high-definition map model.
[0100] In some embodiments, see Figure 7 , step 53 may be the following process:
[0101] Step 531: Obtain a first loss value using the first predicted value and the first true value.
[0102] Step 532: Use the feature extraction network to perform feature extraction on the first prediction value to obtain the first feature output by each layer in the feature extraction network.
[0103] In some embodiments, the feature extraction network can be composed of a VGG-16 network.
[0104] Step 533: Use the feature extraction network to perform feature extraction on the first true value to obtain the second feature output by each layer in the feature extraction network.
[0105] Step 534: Obtain a second loss value using the first feature and the second feature output by each layer.
[0106] Step 535: Use the first loss value and the second loss value to complete auxiliary training of the high-definition map model.
[0107] In some embodiments, the first loss value and the second loss value are calculated in the following manner:
[0108]
[0109] Where L represents the sum of the first loss value and the second loss value, represents the first loss value, represents the second loss value, C represents the number of channels, H represents the image height, that is, the number of pixels in the vertical dimension of the image, W represents the image width, that is, the number of pixels in the horizontal dimension of the image, G represents the first true value, I represents the first predicted value, It means that after the first real value is input into the feature extraction network, the i-th layer output of the feature extraction network is It indicates the output of the i-th layer of the feature extraction network after the first prediction value is input into the feature extraction network.
[0110] In this embodiment, during the training process of the high-definition map model, a road reconstruction map having first real values such as color and / or coordinate information is input into the BEV feature layer in the high-definition map model for prediction, and a first prediction value output by the BEV feature layer is obtained. The first prediction value and the first real value are then used to complete auxiliary training of the high-definition map model, thereby increasing the training data of the high-definition map model and improving the accuracy of subsequent map generation by the high-definition map model. Furthermore, the road reconstruction map has corresponding color and / or coordinate information. During the training process, the high-definition map model can learn information such as color and texture information, thereby improving the accuracy of subsequent map generation by the high-definition map model. Furthermore, the prediction value output by the BEV feature layer is directly used for auxiliary training, thereby avoiding information loss in the decoding stage and supplementing the post-decoding stage of the high-definition map model with rich semantic information.
[0111] In this application, the road reconstruction map is generated offline and is only used in the HD map model training phase. It is not needed during HD map model inference. Therefore, this application will not increase the HD map model inference time while improving the HD map model effect.
[0112] That is, in the process of reasoning the HD map model, the input prediction relationship between the road reconstruction map and the BEV feature layer is not involved. The HD map model reasoning process only involves the above-mentioned encoding layer, BEV feature layer and decoding layer.
[0113] See also Figure 8 , Figure 8 This is a flow chart of an embodiment of a map generation method provided by the present application. The method includes:
[0114] Step 71: Acquire images and / or point clouds.
[0115] Step 72: Input the image and / or point cloud into the high-definition map model to obtain a map output by the high-definition map model.
[0116] The high-definition map model is trained using the method provided in any of the above embodiments.
[0117] In some embodiments, the high-definition map model is not limited to MapTRv2, and any vectorized online mapping model of the encode-decode paradigm can be used.
[0118] In this embodiment, the high-definition map model trained using the method provided in any of the above embodiments can output a more accurate map during the reasoning process.
[0119] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 80 includes a processor 81 and a memory 82 coupled to the processor 81; wherein the memory 82 is used to store a computer program, and the processor 81 is used to execute the computer program to implement the following method:
[0120] Inputting the road reconstruction map into the BEV feature layer to obtain a first prediction value output by the BEV feature layer; wherein the road reconstruction map has a corresponding first true value; and using the first prediction value and the first true value to complete auxiliary training of the high-definition map model;
[0121] Or obtain an image and / or a point cloud; input the image and / or the point cloud into a high-definition map model to obtain a map output by the high-definition map model.
[0122] It can be understood that the processor 81 is also used to execute computer programs to implement the method of any of the above embodiments, which will not be elaborated here.
[0123] See also Fig.10 , Fig.10 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 90 is used to store a computer program 91. When the computer program 91 is executed by a processor, it is used to implement the following method:
[0124] Inputting the road reconstruction map into the BEV feature layer to obtain a first prediction value output by the BEV feature layer; wherein the road reconstruction map has a corresponding first true value; and using the first prediction value and the first true value to complete auxiliary training of the high-definition map model;
[0125] Or obtain an image and / or a point cloud; input the image and / or the point cloud into a high-definition map model to obtain a map output by the high-definition map model.
[0126] It can be understood that when the computer program 91 is executed by the processor, it is also used to implement the method of any of the above embodiments, which will not be elaborated here.
[0127] In summary, the training method, map generation method, device and medium of the high-definition map model provided by the present application, during the training process of the high-definition map model, a road reconstruction map with first real values such as color and / or coordinate information is input into the BEV feature layer in the high-definition map model for prediction, and a first prediction value output by the BEV feature layer is obtained, and then the first prediction value and the first real value are used to complete the auxiliary training of the high-definition map model, thereby increasing the training data of the high-definition map model and improving the accuracy of the subsequent map generation by the high-definition map model. Furthermore, the road reconstruction map has corresponding color and / or coordinate information. During the training process, the high-definition map model can learn information such as color and texture information, thereby improving the accuracy of the subsequent map generation by the high-definition map model. Furthermore, the prediction value output by the BEV feature layer is directly used for auxiliary training, thereby avoiding information loss in the decoding stage and supplementing the high-definition map model with rich semantic information in the post-decoding stage.
[0128] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the circuit or unit is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0130] In addition, each functional unit in each embodiment of the present application may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0131] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for training a high-definition map model, characterized in that: The high-definition map model includes a coding layer, a BEV feature layer and a decoding layer, and the method includes: Inputting a road reconstruction map into the BEV feature layer to obtain a first predicted value output by the BEV feature layer; wherein the road reconstruction map has a corresponding first true value; the first true value includes color and / or coordinates; The first predicted value and the first true value are used to complete auxiliary training of the high-definition map model.
2. The training method according to claim 1, characterized in that: The road reconstruction map is obtained in the following manner: Gridding a sequence of multiple consecutive frames of surround view images at different viewing angles to obtain a first grid image; Optimizing the coordinates and color of each grid point in the first grid image to obtain a second grid image; The second grid image is used as the road reconstruction image, and the coordinates and colors corresponding to the second grid image are used as the first true values.
3. The training method according to claim 2, characterized in that: The step of optimizing the coordinates and color of each grid point in the first grid image to obtain a second grid image includes: Performing image segmentation on the surround view image sequence using a segmentation network to obtain a first image segmentation result; Rendering the first grid image to obtain a two-dimensional image; Using the segmentation network to perform image segmentation on the two-dimensional image to obtain a second image segmentation result; Obtaining a semantic segmentation loss using the first image segmentation result and the second image segmentation result; Obtaining color loss using colors in the two-dimensional image and colors in the surround image sequence; According to the semantic segmentation loss and the color loss, the coordinates and the color of each grid point in the first grid image are optimized to obtain a second grid image.
4. The training method according to claim 3, characterized in that: The step of optimizing the coordinates and the color of each grid point in the first grid image according to the semantic segmentation loss and the color loss to obtain a second grid image includes: Optimizing the intrinsic parameters and extrinsic parameters of an image acquisition device for acquiring the surround view image sequence according to the semantic segmentation loss and the color loss; The coordinates and color of each grid point in the first grid image are adjusted using the optimized internal and external parameters to obtain a second grid image.
5. The training method according to claim 3, characterized in that: In response to the semantic segmentation loss and the color loss meeting a preset condition, stopping optimizing the coordinates and the color of each grid point in the first grid image.
6. The training method according to claim 1, characterized in that: Before inputting the road reconstruction map into the BEV feature layer, the method includes: A rectangular area is determined from the road reconstruction image according to the position of the vehicle, and information corresponding to the rectangular area is used as the first true value.
7. The training method according to claim 1, characterized in that: The using the first predicted value and the first true value to complete the auxiliary training of the high-definition map model includes: Obtaining a first loss value using the first predicted value and the first true value; Performing feature extraction on the first prediction value using a feature extraction network to obtain a first feature output by each layer in the feature extraction network; Performing feature extraction on the first true value using a feature extraction network to obtain a second feature output by each layer in the feature extraction network; The second loss value is obtained by using the first feature and the second feature output by each layer; The first loss value and the second loss value are used to complete auxiliary training of the high-definition map model.
8. A map generation method, characterized in that: The method comprises: Acquire images and / or point clouds; The image and / or the point cloud is input into a high-definition map model to obtain a map output by the high-definition map model; wherein the high-definition map model is trained using the method described in any one of claims 1 to 8.
9. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to the processor; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 8.
