Road element detection model training method and device, detection method and device and electronic equipment

Abstract

The invention relates to a road element detection model training method and device, a detection method and device and electronic equipment, and the training method comprises the steps: obtaining a training sample which comprises local crowdsourcing information corresponding to a target position and foresight semantic information corresponding to the target position, and the local crowdsourcing information and the foresight semantic information are obtained based on a crowdsourcing map; the local crowdsourcing information and the foresight semantic information are input into a road element detection model, a road element vector point set of the target position is obtained through learning, and the road element vector point set represents vectorized road elements and topological information of the road elements; and updating model parameters of the road element detection model according to the road element vector point set. The embodiment of the invention realizes high-precision road detection, does not depend on high-cost equipment, and reduces the cost of automatic driving equipment.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a road element detection model training method, detection method, device and electronic equipment. Background Art

[0002] Detecting road elements, generating vehicle maps based on the detected road elements, and controlling vehicle driving based on the maps are important technologies for realizing autonomous driving. Especially on complex roads, the accuracy of road element detection and the topological relationship of road elements are the safety guarantee for autonomous driving.

[0003] In the related technology, radar sensors are installed on vehicles and combined with high-precision cameras / high-precision maps and other sensing devices to perform high-precision detection of road elements. However, sensing devices such as radar sensors, high-precision cameras and high-precision maps are expensive and have high maintenance costs, which in turn makes the cost of autonomous driving technology high. Summary of the invention

[0004] One of the purposes of the present invention is to provide a method for training a road element detection model to solve the problem that road element detection in the prior art relies on high-cost equipment, resulting in high cost of autonomous driving technology; a second purpose is to provide a method for detecting road elements; a third purpose is to provide a device; and a fourth purpose is to provide an electronic device.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A training method for a road element detection model, the training method comprising:

[0007] Acquire a training sample, wherein the training sample includes local crowdsourcing information corresponding to a target location acquired based on a crowdsourcing map and forward-looking semantic information corresponding to the target location;

[0008] Inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model, learning and obtaining a road element vector point set at the target location, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements;

[0009] The model parameters of the road element detection model are updated according to the road element vector point set.

[0010] According to the above technical means, with local crowdsourcing information and forward-looking semantic information as input, the road element detection model extracts geometric features, semantic features and road topology information of road elements through local crowdsourcing information and forward-looking semantic information. Based on the learning and output of these features, a road element vector point set at the target location can be obtained, and then the model parameters of the road element detection model are updated according to the road element vector point set to obtain a road element detection model suitable for road element detection with local crowdsourcing information and forward-looking semantic information as input. Based on the characteristics of low cost and fast update of crowdsourcing maps, forward-looking semantic information is integrated to obtain rich road information and realize the detection of road elements. Road elements and road topology information are represented by outputting vector points, enriching the direction information of lane lines and lane center lines, providing feasibility reference information such as deceleration and lane change for autonomous driving, and realizing the recognition of road elements based on low-cost perception equipment.

[0011] Furthermore, the inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model to learn and obtain a road element vector point set at the target location includes:

[0012] Inputting the local crowdsourcing information and the forward-looking semantic information into the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information and extracting semantic map features based on the forward-looking semantic information in a feature extractor of the road element detection model, wherein the crowdsourcing map features and the semantic map features reflect road element features of different sizes and / or depths;

[0013] Inputting the crowdsourced map features and the semantic map features into an encoder of the road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on a pre-learned bird's-eye view feature map, and updating the bird's-eye view feature map according to the learning results;

[0014] The updated bird's-eye view feature map is input into the decoder of the road element detection model, and the geometric features, semantic features and road topology information of the road elements at the target location are extracted in the decoder based on the updated bird's-eye view feature map, and the road element vector point set at the target location is output based on the geometric features, the semantic features and the road topology information.

[0015] According to the above technical means, through attention learning, the crowdsourced map features and semantic map features are integrated, and the features of the road element detection model are constructed including geometric features, semantic features and road topology information, so that the road element detection model learns the geometric features, semantic features and road topology information in the road elements corresponding to the crowdsourced map features and the semantic map features, and then learns the road elements about the geometric features, semantic features and road topology information, and forms a road element vector point set. The map is constructed through the road element vector point set to provide direction for vehicle autonomous driving, and the road elements can be distinguished through the vector points.

[0016] Further, the attention learning is performed on the crowdsourced map features and the semantic map features based on the pre-learned bird's-eye view feature map, and the bird's-eye view feature map is updated according to the learning result, including:

[0017] Based on the bird's-eye view feature map obtained by pre-learning, spatial cross-attention interactive learning is performed on the crowdsourcing map features and the semantic map features to obtain cross-attention weights of the crowdsourcing map features and the semantic map features, and the learning result includes the cross-attention weights;

[0018] The bird's-eye view feature map is updated according to the cross attention weights.

[0019] Further, the cross-attention interactive learning of the spatial position of the crowdsourcing map feature and the semantic map feature to obtain the cross-attention weights of the crowdsourcing map feature and the semantic map feature includes:

[0020] Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features;

[0021] The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

[0022] According to the above technical means, the motion state of the object can be inferred and the obstructing object can be identified, thereby inferring the obstructed road elements and improving the accuracy of road element identification.

[0023] Furthermore, the crowdsourced map features and the semantic map features are input into the encoder of the road element detection model, and attention learning is performed on the crowdsourced map features and the semantic map features based on the pre-learned bird's-eye view feature map, including:

[0024] Inputting the crowdsourced map features and the semantic map features into the encoder of the road element detection model, encoding the crowdsourced map features and the semantic map features, and aligning them in spatial position;

[0025] Based on the pre-learned bird's-eye view feature map, attention learning is performed on the crowdsourced map features and semantic map features after encoding alignment.

[0026] According to the above technical means, it is convenient for the encoder to learn crowdsourced map features and semantic map features in spatial position.

[0027] Further, updating the model parameters of the road element detection model according to the road element vector point set includes:

[0028] Obtaining a loss value between an actual road element vector point set at the target location and the road element vector point set in a high-precision map;

[0029] Model parameters of the road element detection model are updated based on the loss value.

[0030] A method for detecting road elements, the method comprising:

[0031] In autonomous driving, local crowdsourced information of a current road traveled by a vehicle is obtained based on a crowdsourced map, and forward-looking semantic information of the vehicle on the current road traveled is obtained;

[0032] The local crowdsourcing information and the forward-looking semantic information are input into a road element detection model to obtain a road element vector point set of the current driving road, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements, wherein the road element detection model is obtained by the training method described above.

[0033] According to the above technical means, in autonomous driving, with local crowdsourcing information and forward-looking semantic information as input, the road element detection model extracts geometric features, semantic features and road topology information of road elements through local crowdsourcing information and forward-looking semantic information. Based on the learning and output of these features, the road element vector point set of the current driving road can be obtained to control the vehicle's autonomous driving through the road element vector point set. Based on the low cost and fast update characteristics of crowdsourcing maps, forward-looking semantic information is integrated to obtain rich road information, realize the detection of road elements, and characterize road elements and road topology information by outputting vector points, enriching the direction information of lane lines and lane center lines, providing feasibility reference information such as deceleration and lane change for autonomous driving, and realizing the recognition of road elements based on low-cost perception equipment.

[0034] Furthermore, the inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model to obtain a road element vector point set of the current driving road includes:

[0035] Inputting the local crowdsourcing information and the forward-looking semantic information into the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information and extracting semantic map features based on the forward-looking semantic information in a feature extractor of the road element detection model, wherein the crowdsourcing map features and the semantic map features reflect road element features of different sizes and / or depths;

[0036] Inputting the crowdsourced map features and the semantic map features into an encoder of the road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on a bird's-eye view feature map in the road element detection model, and obtaining a target bird's-eye view feature map according to the learning results;

[0037] The target bird's-eye view feature map is input into the decoder of the road element detection model, in which the geometric features, semantic features and road topology information of the road elements of the current driving road are extracted based on the target bird's-eye view, and the road element vector point set of the current driving road is output based on the geometric features, the semantic features and the road topology information.

[0038] Further, the method of performing attention learning on the crowdsourced map features and the semantic map features based on the bird's-eye view feature map in the road element detection model, and obtaining a target bird's-eye view feature map according to the learning result, includes:

[0039] Based on the bird's-eye view feature map in the road element detection model, spatial cross-attention interactive learning is performed on the crowd-sourced map features and the semantic map features to obtain cross-attention weights of the crowd-sourced map features and the semantic map features, wherein the learning result includes the cross-attention weights;

[0040] A target bird's-eye view feature map is obtained according to the cross attention weights.

[0041] Further, the crowdsourcing map features and the semantic map features are subjected to spatial cross-attention interactive learning to obtain cross-attention weights of the crowdsourcing map features and the semantic map features, and the learning results include the cross-attention weights, including:

[0042] Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features;

[0043] The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

[0044] A training device for a road element detection model, the training device comprising:

[0045] A first acquisition module is used to acquire a training sample, wherein the training sample includes local crowdsourcing information corresponding to a target location acquired based on a crowdsourcing map and forward-looking semantic information corresponding to the target location;

[0046] A detection model, used for learning and obtaining a road element vector point set at the target location according to the input of the local crowdsourcing information and the forward-looking semantic information, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements;

[0047] The training module is used to update the model parameters of the detection model according to the road element vector point set to obtain a road element detection model.

[0048] A road element detection device, the detection device comprising:

[0049] A second acquisition module is used to acquire local crowdsourced information of the road on which the vehicle is currently traveling based on the crowdsourced map, and is also used to acquire forward-looking semantic information of the vehicle on the road on which the vehicle is currently traveling;

[0050] A road element detection model is used to learn and obtain a road element vector point set of the current driving road based on the input of the local crowdsourcing information and the forward-looking semantic information, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements.

[0051] An electronic device, comprising: a memory and a processor;

[0052] The memory is used to store computer programs / instructions; the processor is used to implement the training method as described above, or implement the detection method as described above according to the computer programs / instructions stored in the memory.

[0053] A computer-readable storage medium stores a computer program / instruction, wherein the computer program / instruction is used to implement the method described above when executed by a processor.

[0054] A computer program product, comprising a computer program / instruction, wherein the computer program / instruction implements the method as described above when executed by a processor.

[0055] Beneficial effects of the present invention:

[0056] (1) In the embodiment of the present invention, the road element detection model uses local crowdsourcing information and forward-looking semantic information as input, and extracts geometric features, semantic features, and road topology information of road elements through local crowdsourcing information and forward-looking semantic information. Based on the learning and output of these features, a road element vector point set at the target location can be obtained, and then the model parameters of the road element detection model are updated according to the road element vector point set to obtain a road element detection model suitable for detecting road elements using local crowdsourcing information and forward-looking semantic information as input. The crowdsourcing map has the characteristics of low cost and fast update, and integrates forward-looking semantic information to obtain rich road information and realize the detection of road elements. By outputting vector points to represent road elements and road topology information, the direction information of lane lines and lane center lines is enriched, and feasibility reference information such as deceleration and lane change is provided for autonomous driving, realizing the recognition of road elements based on low-cost perception equipment.

[0057] (2) The road element detection model learns the geometric features, semantic features, and road topology information of road elements. By learning the road elements at the intersection, it can generate an intersection map with lane centerlines and lane lines, so that when the vehicle automatically drives to the intersection, it can timely perform safety control on the lane lines at the intersection. The lane centerline and road topology information can provide direction information for the vehicle. At roundabouts or some intersections without lane lines, the lane centerline and road topology information can provide the vehicle with the driving lane and direction.

[0058] (3) Crowdsourcing data has the characteristics of a wide range, while forward-looking semantic information is real-time. By interactively learning crowdsourcing information and forward-looking semantic information, the extension and interaction of road elements can be inferred based on the extensive crowdsourcing data. Based on the forward-looking semantic information, the road information of the current driving road can be updated in time to ensure that the road element vector point set obtained is a more comprehensive and timely road element, thereby improving the accuracy of road element detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of the structure of the road element detection model proposed in the embodiment of the present application;

[0060] Figure 2 A schematic diagram of a process flow of a road element detection model training method provided in an embodiment of the present application;

[0061] Figure 3 for Figure 2 A detailed flow chart of step S202;

[0062] Figure 4a A road element vector point set detected by a road element detection model provided in an embodiment of the present application in a road scene;

[0063] Figure 4b A road element vector point set detected by the road element detection model provided in an embodiment of the present application in another road scene;

[0064] Figure 4c A road element vector point set detected by the road element detection model provided in an embodiment of the present application in another road scene;

[0065] Figure 5 A schematic diagram of a process flow of a road element detection method provided in an embodiment of the present application;

[0066] Figure 6 for Figure 5 A detailed flow chart of step S502;

[0067] Figure 7 A comparison chart of the detection results of the road element detection method provided in an embodiment of the present application and the effect of the crowdsourced map. DETAILED DESCRIPTION

[0068] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0069] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0070] Autonomous driving technology is an important development direction of the current automotive industry. It enables cars to autonomously perceive the environment, make decisions and perform driving tasks through advanced sensors, controllers, actuators, computing platforms and communication technologies, achieving safe and efficient operation. Autonomous perception of the environment includes detecting road elements, obtaining the road environment through the detected road elements, and then controlling the vehicle to perform autonomous driving tasks according to the road environment.

[0071] The road element detection accuracy and the topological relationship of road elements are important factors affecting the safety performance of autonomous driving, especially on complex roads. Improving the road element detection accuracy and enhancing the road topological relationship are issues that need to be urgently addressed in autonomous driving technology.

[0072] At present, the research on the recognition and detection of road elements is mostly about how to upgrade from the camera plane to three-dimensional space and overcome projection errors. With the popular trend of BEV perception, segmentation and vectorization methods are gradually adopted to learn high-precision maps. However, most of these methods rely on perception devices such as radar sensors, high-precision cameras / high-precision maps to perform high-precision detection of road elements. Perception devices such as radar sensors, high-precision cameras and high-precision maps are expensive and have high maintenance costs, which in turn increases the cost of autonomous driving technology, making it difficult to apply autonomous driving technology to low-cost vehicles.

[0073] Based on this, the embodiment of the present application provides a method for detecting road elements. The method uses crowdsourcing data as a feature layer to learn and update based on deep learning, and outputs vectorized road elements. Vectorized road elements can provide better support for downstream vehicle control. Crowdsourcing maps rely on low-cost camera generation, and forward-looking semantic maps can be generated based on forward-looking cameras, realizing the collection of road elements using low-cost resources. By combining the large-scale data contained in the crowdsourcing data and the small-scale data contained in the forward-looking semantic map, and combining map data of different depths and sizes, more accurate road element information can be obtained.

[0074] This application describes the deep learning process and road element detection process in detail through the following embodiments.

[0075] Figure 1 This is a schematic diagram of the structure of the road element detection model proposed in the embodiment of the present application. Figure 1 As shown, it includes a feature extractor Backbon), an encoder Transformer-Encode) and a decoder Transformer-Decode). Among them:

[0076] Feature extractors are used for feature extraction. For example, Figure 1 In this example, the crowdsourced map urb map is located and intercepted to obtain a small map of local crowdsourced information, which is then input into the road element detection model. In addition, the forward-looking semantic information of the corresponding position is obtained through the vehicle's forward-looking camera and input into the road element detection model. The feature extractor extracts crowdsourced map features from local crowdsourced information and semantic map features from forward-looking semantic information, and inputs them into the encoder.

[0077] The encoder is used to encode crowdsourced map features and semantic map features, and use crowdsourced map features and semantic map features as keys and values ​​to perform spatial or temporal attention learning with the BEV feature map to learn a new BEV feature map, which is based on the fusion of crowdsourced map features and semantic map features and output from the BEV perspective.

[0078] The decoder is used to extract features from the new BEV feature map to obtain a road element vectorized point set.

[0079] Based on the structure of the above model, an embodiment of the present application provides a method for training a road element detection model, and its training process is described in detail through the following embodiments.

[0080] Figure 2 This is a schematic diagram of the structure of the road element detection model proposed in the embodiment of the present application. Figure 2 As shown, the training method includes:

[0081] S201, obtaining a training sample, where the training sample includes local crowdsourcing information corresponding to a target location obtained based on a crowdsourcing map and forward-looking semantic information corresponding to the target location;

[0082] The road element detection model proposed in the embodiment of the present application uses crowdsourced maps and forward-looking semantic information collected by ordinary cameras in the vehicle's forward-looking perspective as inputs to perform road element detection learning. Among them, the crowdsourced map is obtained by cloud-based fusion of a large amount of data uploaded from multiple vehicles. The camera on each vehicle collects image data and uploads it to the cloud. The cloud obtains the crowdsourced map based on the fusion of a large amount of image data. Therefore, the crowdsourced map has the advantages of a wide range, timely updates, and can be obtained offline. In addition, the crowdsourced map does not have high requirements for the cameras installed on the vehicle, and the vehicle can be equipped with low-cost cameras. The camera with a forward-looking perspective has the advantage of collecting small-range data with a deeper depth, which can make up for the detailed road elements in the crowdsourced map.

[0083] In this step, the crowdsourced map needs to go through multiple rounds of map fusion and splicing, and multiple fusions by multiple vehicles to ensure that the crowdsourced map has a high quality.

[0084] In this step, the training samples are obtained in the following ways, including but not limited to:

[0085] Take a set of training samples as an example, combined with reference Figure 1 :

[0086] Select a target location (to detect road elements at an intersection, select the target intersection area as the target location), and build a global crowdsourcing map urb_map through image data uploaded by multiple vehicles. The static elements in urb_map must be consistent with the static elements output by the road element detection model.

[0087] By locating the vehicle position in urb_map, the local point cloud map small_map (local crowdsourcing information) with the vehicle as the origin or a specific point is intercepted according to the point cloud range pc_range.

[0088] The forward-looking semantic information of the target position collected in advance by the forward-looking camera of the vehicle is selected, and the small_map and the forward-looking semantic information are used as a set of training samples as the input of the road element detection model. Optionally, the forward-looking semantic information is formed by splicing multiple frames of forward-looking semantic maps through slam.

[0089] Multiple groups of training samples are obtained in the above manner, and the multiple groups of training samples are input into the road element detection model to perform multiple groups of sample training.

[0090] As an example, the training sample is a training sample based on a bird's-eye view obtained by preprocessing. Specifically, the local crowdsourcing data corresponding to the target location is obtained through the crowdsourcing map, and then the local crowdsourcing data is parsed and projected to obtain the local crowdsourcing information corresponding to the bird's-eye view. Similarly, the forward-looking map of the target location is collected by the vehicle's forward-looking camera, the forward-looking map is spliced, and the spliced ​​forward-looking map is projected to the bird's-eye view to obtain the forward-looking semantic information corresponding to the bird's-eye view.

[0091] S202, inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model, learning and obtaining a road element vector point set at a target location, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements;

[0092] In an embodiment of the present application, the road element detection model is configured to learn the geometric features, semantic features and road topology information of road elements. The geometric features, semantic features and road topology information of road elements are expressed using a vector point set, where each vector point corresponds to a coordinate value. The direction of the road element can be determined based on the coordinate value, and the classification or type of the road element can be determined based on the semantic features.

[0093] When constructing a road element detection model, the features of the road element detection model are constructed based on the geometric features, semantic features and road topology information of the above-mentioned road elements. Among them, the geometric features of the road elements include lane center lines, lane lines, road edges, sidewalks, etc. Each line segment consists of an ordered set of N points, and road elements with different geometric features are distinguished by points of different colors and / or ordered sets of points with field labels. The semantic features of road elements include lane classification (sidewalk, lane) and lane line type (dashed line, solid line, virtual line). Road topology information includes an adjacency matrix reflected by the connection of lane lines. Based on the constructed features, when extracting features, the model extracts and learns according to the constructed features. The road element detection model outputs a road element vector point set at the target location, which reflects the geometric features, semantic features and road topology information of the road elements.

[0094] In this step, after the road element detection model is constructed, the local crowdsourcing information and forward-looking semantic information are input into the road element detection model. The road element detection model learns based on the local crowdsourcing information and forward-looking semantic information to obtain a road element vector point set at the target location.

[0095] S203: Update model parameters of the road element detection model according to the road element vector point set.

[0096] The road element vector point set is learned based on the current model parameters of the road element detection model. The current model parameters affect the accuracy of the output road element vector point set. Therefore, it is necessary to continuously learn different training samples and continuously adjust the model parameters to obtain a road element detection model with higher detection accuracy.

[0097] In this step, by judging the loss of the road element vector point set relative to the true value, it is determined whether the model parameters need to be adjusted and the adjustment value of the model parameters, and then the model parameters are adjusted. When the local crowdsourcing information and forward-looking semantic information are input to the road element detection model again, the road elements are continuously learned based on the adjusted model parameters, so that the road element detection model can learn accurate road element information.

[0098] As an example, a high-precision map can be used as a reference, and the road element detection model can be trained based on the actual road element vector point set corresponding to the target position in the high-precision map as the true value. Optionally, the specific implementation of S203 can be:

[0099] Obtaining the loss value between the actual road element vector point set and the road element vector point set at the target location in the high-precision map;

[0100] Model parameters of the road element detection model are updated based on the loss value.

[0101] As an example, the model parameters of the road element detection model are updated based on the loss value until the loss value is less than or equal to a preset threshold, and then it is determined that the road element detection model training is completed.

[0102] As an example, the model parameters of the road element detection model are updated based on the loss value until the number of learning times reaches a preset number, and then it is determined that the road element detection model training is completed.

[0103] Specifically, the input local crowdsourcing information and forward-looking semantic information are extracted through the Transformer network, and the loss is calculated with the true value of the road elements obtained from the high-precision map HDmap, and then the network weights and feature layers are updated through back-propagation iterations.

[0104] Alternatively, whether the road element detection model training is completed may be determined in other ways.

[0105] In the example taking the high-precision map as the true value, since the expression form of the road elements in the high-precision map is different from the expression form of the feature point output constructed in this embodiment, in order to facilitate comparison and improve the accuracy of the loss value calculation, before calling the high-precision map for model training, the high-precision map needs to be preprocessed so that the expression form of the road elements in the high-precision map is consistent with the expression form of the road elements constructed in the embodiment itself.

[0106] To give an example, the sidewalk feature in the high-precision map is a closed rectangle composed of 5 points, while the sidewalk output in the embodiment of the present application is a lane line composed of multiple points. In this case, the sidewalk rectangle frame needs to be processed into the form of a lane line according to the direction of the vehicle, and the number N of ordered collection points that constitute the line segment should be consistent with the lane line, curb, etc.

[0107] The model training outputs vectorized coordinate points, so the lane line samples of each frame of the high-precision map need to be resampled to ensure that the dimension of each lane instance output is the same. The resampling method is to uniformly sample N points based on the original lane length.

[0108] As an example, the actual road element vector point set at the target position, ie, the above-mentioned true value, may also be determined based on other methods.

[0109] The embodiment of the present application adopts a segmented approach to learn and predict the lane centerline and centerline topology. The lane centerline adopts position constraint fitting coordinate points, and the centerline topology constructs the connection relationship into an adjacency matrix. The connection relationship of the adjacency matrix is ​​predicted by updating the gradient transfer of the deep learning network.

[0110] In the embodiment of the present application, the road element detection model uses local crowdsourcing information and forward-looking semantic information as input to extract geometric features, semantic features, and road topology information of road elements through local crowdsourcing information and forward-looking semantic information. Based on the learning and output of these features, a road element vector point set at the target location can be obtained, and then the model parameters of the road element detection model are updated according to the road element vector point set to obtain a road element detection model suitable for road element detection using local crowdsourcing information and forward-looking semantic information as input. Based on the characteristics of low cost and fast update of crowdsourcing maps, forward-looking semantic information is integrated to obtain rich road information and realize the detection of road elements. Road elements and road topology information are represented by outputting vector points, enriching the direction information of lane lines and lane center lines, providing feasibility reference information such as deceleration and lane change for autonomous driving, and realizing the recognition of road elements based on low-cost perception equipment.

[0111] As an example, see Figure 3 Based on the above embodiment, the specific implementation of step S202 may be:

[0112] S301, inputting local crowdsourcing information and forward-looking semantic information into a road element detection model, and in a feature extractor of the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information, and extracting semantic map features based on the forward-looking semantic information;

[0113] Among them, the crowdsourced map features and the semantic map features reflect different sizes and / or depths of road element features;

[0114] Combination Figure 1 The structure of the road element detection model shown in the embodiment of the present application is based on the data from the crowdsourced map and the data collected from the vehicle's front-view camera as learning objects input into the road element detection model. Therefore, when the local total package information small map and the forward-looking semantic information are input into the road element detection model together, the feature extractor backbone of the road element detection model needs to extract features of data from different sources, that is, to perform feature extraction on the local total package information and the forward-looking semantic information respectively, and obtain the crowdsourced map features corresponding to the local crowdsourced information and the semantic map features corresponding to the forward-looking semantic information.

[0115] The small map comes from the crowdsourced map and is responsible for offline perception of large-scale map information. The semantic map comes from the semantic road information perceived by the vehicle's forward vision. The feature scales and / or depths of the small map and the forward-looking semantic information are different. When selecting training samples, the target position is sampled according to the scale ratios corresponding to the two types of information to obtain feature maps of two different scales and / or depths. The feature extractor can extract crowdsourced map features and semantic map features of different scales and / or depths based on the small map and forward-looking semantic information.

[0116] Optionally, the crowdsourced map features include at least one of geometric features, semantic features and road topology information of road elements; and the forward-looking semantic map features include at least one of geometric features, semantic features and road topology information of road elements.

[0117] S302, inputting the crowdsourced map features and the semantic map features into the encoder of the road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on the pre-learned bird's-eye view feature map, and updating the bird's-eye view feature map according to the learning results;

[0118] Optionally, the learning result refers to the fusion result of the crowdsourced map features and the semantic map features in the BEV space.

[0119] In this step, the road element detection model adopts the BV-Transformer structure, which combines the bird's-eye-view (BEV) perspective and the Transformer model to achieve more accurate and efficient road perception. Based on the BEV perspective, crowdsourced map features and semantic map features are converted to the BEV space, thereby overcoming the scale and occlusion problems under the traditional 2D image perspective. The Transformer model is a neural network model based on the attention mechanism, which can handle inputs of different lengths and structures. In the process of learning road elements, the road element detection model in this step can fuse crowdsourced map features and semantic map features of different scales and depths in the BEV space, perform perception fusion in advance, and realize temporal fusion, making the perception effect more stable and accurate.

[0120] Before the road element detection model is trained, the bird's-eye view feature map is a pre-set blank bird's-eye view feature map. During the road element detection model training process, the crowdsourced map features and semantic map features are spatially aligned and fused with the blank bird's-eye view map to obtain the target bird's-eye view feature map. The target bird's-eye view map is the BEV expression after the fusion of crowdsourced map features and semantic map features. Compared with the blank bird's-eye view feature map, the target bird's-eye view feature map has stronger expression ability and expresses road elements more clearly. In order to improve the detection accuracy of the road element detection model, the blank bird's-eye view feature map is updated based on the learned target bird's-eye view feature map, which is defined as the pre-learned bird's-eye view feature map. During the model training process, the pre-learned bird's-eye view feature map is continuously updated based on the target bird's-eye view feature map until the end of the model training, and the last updated bird's-eye view feature map is used as the bird's-eye view feature map of the road element detection model.

[0121] Therefore, the crowdsourced map features and semantic map features are input into the encoder of the road element detection model. If it is the first input, attention learning is performed on the crowdsourced map features and semantic map features based on a preset blank bird's-eye view feature map, and then the blank bird's-eye view feature map is updated based on the learning results.

[0122] For example, when the road element detection model starts training, the crowdsourced map features and semantic map features are input into the encoder for the first time. The encoder fuses the crowdsourced map features and semantic map features in the blank bird's-eye view feature map to obtain a target bird's-eye view feature map that fuses the crowdsourced map features and semantic map features. The target bird's-eye view feature map has a better learning effect on road elements than the blank bird's-eye view feature map. Based on this, the target bird's-eye view feature map is used to update the blank bird's-eye feature map, that is, the bird's-eye feature map is continuously learned during the road element detection model training process. When the crowdsourced map features and semantic map features are input into the encoder for the Nth time, the encoder fuses the crowdsourced map features and semantic map features in the bird's-eye view feature map learned from the previous training sample to obtain a target bird's-eye view feature map that fuses the crowdsourced map features and semantic map features. The bird's-eye view feature map learned from the previous training sample is continuously updated based on the target bird's-eye feature map until the model training is completed, where N ≥ 2.

[0123] Optionally, taking the weight matrix of the road elements in the BEV space reflected by the bird's-eye view feature map as an example, the updating process of the bird's-eye view feature map is explained: assuming that the current pre-learned weight matrix is ​​W1, in this learning, the target bird's-eye view feature map is obtained through crowdsourcing map features, semantic map features and the weight matrix W1, and the weight matrix of the target bird's-eye view feature map reflecting the road elements in the BEV space is W2. Based on the weight matrix W2 of the target bird's-eye view feature map, the pre-learned bird's-eye view feature map is updated. In this way, in the next training, the weight matrix of the road elements in the BEV space reflected by the pre-learned bird's-eye view feature map is W2.

[0124] As an example, based on the pre-learned bird's-eye view feature map, attention learning is performed on crowdsourced map features and semantic map features, and the bird's-eye view feature map is updated according to the learning results, including:

[0125] Based on the pre-learned bird's-eye view feature map, the crowdsourcing map features and the semantic map features are interactively learned through spatial cross-attention, and the cross-attention weights of the crowdsourcing map features and the semantic map features are obtained.

[0126] Update the bird's-eye view feature map according to the criss-cross attention weights.

[0127] In this example, the cross-attention mechanism is used to learn crowdsourcing map features and semantic map features. The learning of cross-attention weights is taken as an example for detailed explanation.

[0128] In this example, crowdsourcing map features and semantic map features are used as Key and Value, and BEV feature map is used as Query. By calculating the attention score between Query and Key, the attention score is normalized by Softmax to obtain the attention weight, and the Value is weighted and summed using the attention weight, and then fused to obtain the output - the target bird's-eye view feature map.

[0129] Through the cross-attention mechanism, the model can spatially align and fuse crowdsourced map features and semantic map features with the BEV feature map, thereby enhancing the expressiveness of the BEV feature map.

[0130] Optionally, the learning results include but are not limited to cross-attention weights, and may also include scale or depth, local information and global information, etc.

[0131] In this example, by performing spatial cross-attention interactive learning on crowdsourced map features and semantic map features, different cross-attention weights can be assigned to road element features of different sizes or depths, and then fused based on the cross-attention weights to obtain more comprehensive and higher-precision road elements, enriching the output road elements.

[0132] As an example, the crowdsourcing map features and the semantic map features are interactively learned by cross-attention in spatial position, and the cross-attention weights of the crowdsourcing map features and the semantic map features are obtained, including:

[0133] Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features;

[0134] The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

[0135] In this example, the cross-attention mechanism and self-attention mechanism are used to learn crowdsourcing map features and semantic map features.

[0136] Among them, the crowdsourced map features of the current time frame and the crowdsourced map features of the historical time frame are determined based on the time sequence, the self-attention score between the crowdsourced map features of the current time frame and the crowdsourced map features of the historical time frame is calculated, and the crowdsourced map features of the historical time frame are weighted summed based on the attention to obtain the fused crowdsourced map features. Similarly, the semantic map features of the current time frame and the semantic map features of the historical time frame are determined based on the time sequence, the self-attention score between the semantic map features of the current time frame and the semantic map features of the historical time frame is calculated, and the weighted sum of the semantic map features of the historical time frame is obtained based on the attention to obtain the fused semantic map features.

[0137] In this example, based on the feature fusion of the previous and next time frames, the motion trajectory and dynamic changes of the object can be captured, so as to better infer the motion state of the object and identify the occluded objects, and then more accurately identify the road elements, thereby improving the accuracy of road element detection.

[0138] Furthermore, the fused crowdsourcing map features and fused semantic map features are used as Key and Value, and the BEV feature map is used as Query. By calculating the attention score between Query and Key, and performing weighted summation on Value according to the attention score, the fused bird's-eye view feature map is obtained.

[0139] As an example, based on the pre-learned bird's-eye view feature map, interactive attention learning of the spatial position of the crowdsourced map features can be performed to obtain interactive attention weights for the crowdsourced map features and the semantic map features, and the first bird's-eye view feature map can be output according to the interactive attention weights.

[0140] Self-attention interactive learning is performed on at least two adjacent frames of the first bird's-eye view feature map at the temporal level, a second bird's-eye view feature map is output according to the self-attention learning weight, and the pre-learned bird's-eye view feature map is updated based on the second bird's-eye view feature map.

[0141] Specifically, crowdsourced map features and semantic map features are used as Key and Value, and BEV feature map is used as Query. By calculating the attention score between Query and Key, and weighted summing the Value according to the attention score, the first bird's-eye view feature map is obtained. Then, based on the temporal characteristics of each first bird's-eye view feature map, the adjacent first bird's-eye view feature maps are interactively fused with self-attention, and the physical motion trajectory and dynamic changes are learned by combining the current first bird's-eye view feature map and the historical first bird's-eye view feature map, and the self-attention weights of the road elements are adjusted, and then the second bird's-eye view feature map is output, and the pre-learned bird's-eye view feature map is updated for the second bird's-eye view feature map until the model training is completed.

[0142] This example can fuse road element features of different sizes or depths by performing spatial cross-attention interactive learning on crowdsourced map features and semantic map features to obtain more comprehensive and higher-precision road elements and enrich the output road elements; combined with the self-attention interactive learning of crowdsourced map features and semantic map features at the temporal level, it can combine crowdsourced map features and semantic map features of multiple time frames to capture the movement trajectory and dynamic changes of objects, thereby better inferring the movement state of objects and identifying occluded objects, and then inferring that the correct ones are road elements, making the road element vector point set output by the model more accurate.

[0143] As an example, after the crowdsourced map features and the semantic map features are input into the encoder of the road element detection model, the crowdsourced map features and the semantic map features are first spatially encoded and aligned; then, based on the pre-learned bird's-eye view feature map, attention learning is performed on the aligned crowdsourced map features and the semantic map features.

[0144] Since crowdsourced map features and semantic map features are extracted from local crowdsourced information and forward-looking semantic information, respectively, local crowdsourced information is obtained from crowdsourced maps, and forward-looking semantic information is obtained from forward-looking cameras, the spatial positions of the same road elements may be different due to different information sources. Based on this, before learning crowdsourced map features and semantic map features, this embodiment first encodes crowdsourced map features and semantic map features, and performs spatial position alignment based on the encoding. For example, the encoding corresponding to the same position is the same, and the positions of crowdsourced map features and semantic map features are aligned based on the same encoding.

[0145] The crowdsourced map features and the semantic map features are spatially aligned to facilitate the encoder's learning of the same positions of the crowdsourced map features and the semantic map features, thereby improving the learning accuracy.

[0146] S303, input the updated bird's-eye view feature map into the decoder of the road element detection model, extract the geometric features, semantic features and road topology information of the road elements at the target location based on the updated bird's-eye view feature map in the decoder, and output the road element vector point set at the target location based on the geometric features, semantic features and road topology information.

[0147] After obtaining the bird's-eye view feature map that integrates crowdsourced map features and semantic map features (updated bird's-eye view feature map), the bird's-eye view feature map is input into the decoder Transformer-Decoder, which further extracts the geometric features, semantic features and road topology information of the deep road from the bird's-eye view feature map. The geometric features are represented as line segments consisting of the center line and its corresponding lane boundary. Each line segment is defined as an ordered set of N points in 3D space. Semantic features include lane classification (sidewalk, lane) and lane line type (dashed line, solid line, virtual line). The road topology information is a lane topology map of G = (V, E), where the node V of the topology map represents the lane segment, the edge E of the topology map represents the connectivity between the lane segments, and the lane connection relationship represented by the adjacency evidence.

[0148] In an embodiment of the present application, the crowdsourced map can obtain a wider field of view, and the forward-looking semantic map can obtain real-time perception information to timely update the road elements on the current road. The crowdsourced information and the forward-looking semantic information are combined, and the road element detection accuracy is improved based on the wide field of view and the close-range information supplement.

[0149] After completing the training of the road element detection model through the above training method, local crowdsourcing information corresponding to a straight road, a road with a U-turn intersection, and a certain intersection and the forward-looking semantic information collected by the vehicle at these locations are collected as test samples, and the trained road element detection model is tested to verify the learning effect of the road element detection model.

[0150] like Figure 4a As shown, the local package information and forward-looking semantic information corresponding to the straight road are input into the road element detection model, and the road detection model outputs the road element vector point set corresponding to the straight road. In the figure, different colors are used to represent road elements, for example, purple is the center line of the lane, and the lane lines are on both sides of the center line of the lane.

[0151] As shown in FIG4B , the local package information and forward-looking semantic information corresponding to the road with a U-turn intersection are input into the road element detection model, and the road detection model outputs the road element vector point set corresponding to the straight road. In the figure, the lane centerline and lane line can be output in the U-turn intersection. In addition, the road elements are represented by different colors, and the lane lines are classified according to the density of the point set.

[0152] As shown in Figure 4C, the local package information and forward-looking semantic information corresponding to a certain intersection are input into the road element detection model, and the road detection model outputs the road element vector point set corresponding to the straight road. In the figure, different colors are used to represent road elements, for example, purple is the lane centerline, and the lane lines are on both sides of the lane centerline; the lane centerline is output at the intersection, and the lane is inferred based on the lane centerline, which is convenient for vehicle automatic driving control.

[0153] After the road element detection model is trained through the above training method, the road element detection model is deployed on the orin platform and tested on the road. In the process of converting the pth to onnx and tensorrt models, the reference point reference_point in the transformer operator is updated and transmitted in layers. In the orin platform deployment process, the trained road element detection model pth file is converted to onnx, and then converted to an engine file through tensorrt. After post-processing and compilation into a library file on the orin platform, it can be run on the vehicle.

[0154] Based on the application of the road element detection model on a real vehicle, an embodiment of the present application also provides a road element detection method to realize the detection of road elements during the vehicle's autonomous driving process.

[0155] Figure 5 A schematic diagram of a road element detection method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the detection method includes:

[0156] S501, in the autonomous driving, obtaining local crowdsourced information of the road the vehicle is currently traveling on based on the crowdsourced map, and obtaining forward-looking semantic information of the vehicle on the road the vehicle is currently traveling on;

[0157] As an example, the crowdsourced map can be obtained through the cloud, and can be an offline crowdsourced map or can be downloaded online. The front-view semantic information is obtained based on the front-view image collected in real time by the front-view camera on the vehicle.

[0158] Since the crowdsourcing map has a wide range, and during the vehicle's autonomous driving process, it is only necessary to detect road elements within a certain range of the vehicle. Therefore, the vehicle can be positioned to obtain the vehicle's current driving road, and then the local crowdsourcing information corresponding to the current driving road can be intercepted. At the same time, the forward-looking semantic image currently captured by the vehicle's front-view camera (which is also the forward-looking semantic image on the vehicle's current driving road) can be obtained to obtain the forward-looking semantic information.

[0159] As an example, the forward-looking semantic information is spliced ​​through SLAM based on multiple frames of forward-looking semantic maps of the vehicle. The multi-frame spliced ​​forward-looking semantic information can enable the model to obtain richer semantic information.

[0160] S502, inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model to obtain a road element vector point set of the current driving road, where the road element vector point set represents the vectorized road elements and the topological information of the road elements;

[0161] The road element detection model is obtained through the training methods of the above-mentioned embodiments.

[0162] After the road element detection model is trained, the road element vector point set corresponding to the current driving road can be output directly based on the input information and the model parameters of the road element detection model. The specific implementation process can refer to the examples listed in the following embodiments.

[0163] In this embodiment, based on the pre-trained road element detection model, it is applied to the autonomous driving technology. During the vehicle driving process, the road element vector point set can be calculated through the crowdsourced map and forward-looking semantic information. The road vector point set represents the road elements and the topological information of the road elements, wherein the road elements include the center line, lane line, pedestrian crossing, zebra crossing, etc. The vehicle can perform autonomous driving control through the road element vector point set, and it is not dependent on high-cost equipment and can also realize the detection of road elements.

[0164] As an example, see Figure 6 Based on the above embodiment, the specific implementation of step S502 may be:

[0165] S601, inputting local crowdsourcing information and forward-looking semantic information into a road element detection model, and in a feature extractor of the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information, and extracting semantic map features based on the forward-looking semantic information;

[0166] Among them, the crowdsourced map features and the semantic map features reflect different sizes and / or depths of road element features.

[0167] In this step, the process and method of feature extraction are the same as above. Figure 3 The extraction process and method are the same as in , so I will not go into details here.

[0168] S602, inputting the crowdsourced map features and the semantic map features into an encoder of a road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on a bird's-eye view feature map in the road element detection model, and obtaining a target bird's-eye view feature map according to the learning result;

[0169] As an example, the bird's-eye view feature map in the road element detection model is obtained by continuous learning and updating during the training process of the road element detection model. The target bird's-eye view feature map is obtained by the road element detection model using the bird's-eye view feature map to pay attention to crowdsourced map features and semantic map features. The target bird's-eye view feature map is an output parameter of the road element detection model.

[0170] For example, assume that the bird's-eye view feature map in the road element detection model reflects the weight matrix w1 of the road element in the BEV space; the crowdsourced map feature T1 and the semantic map feature T1 are input into the decoder of the road element detection model, and the output target bird's-eye view feature map is: [T1, T2]*weight matrix w1.

[0171] As an example, the cross-attention learning mechanism is used to learn crowdsourced map features and semantic map features as follows:

[0172] Perform spatial cross-attention interactive learning on crowdsourced map features and semantic map features to obtain the attention weights of crowdsourced map features and semantic map features. The learning results include the attention weights.

[0173] Obtain the target bird's-eye view feature map according to the attention weight.

[0174] In this example, the learning process and principle of the road element detection model for crowdsourced map features and semantic map features are similar to the above Figure 3 The exemplary embodiments are the same and will not be described again here.

[0175] As an example, the cross-attention learning mechanism and the self-attention learning mechanism are used to learn crowdsourced map features and semantic map features, as follows:

[0176] Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features;

[0177] The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

[0178] In this example, the learning process and principle of the road element detection model for crowdsourced map features and semantic map features are similar to the above Figure 3 The exemplary embodiments are the same and will not be described again here.

[0179] It should be noted that whether to first perform spatial cross-attention learning on the crowdsourced map features and the semantic map features, and then perform self-attention learning at the temporal level based on the BEV feature map after spatial cross-attention learning, or to first perform self-attention learning at the temporal level on the crowdsourced map features and the semantic map features separately, and then perform spatial cross-attention learning on the crowdsourced map features and the semantic map features, depends on the method adopted in the training process of the road element detection model, and is not limited to one method.

[0180] S603, input the target bird's-eye view feature map into the decoder of the road element detection model, extract the geometric features, semantic features and road topology information of the road elements of the current driving road based on the target bird's-eye view in the decoder, and output the road element vector point set of the current driving road based on the geometric features, semantic features and road topology information.

[0181] After obtaining the bird's-eye view feature map that integrates crowdsourced map features and semantic map features, the bird's-eye view feature map is input into the decoder Transformer-Decoder. The decoder further extracts deep road geometric feature vectorization information, road semantic feature vectorization information, and road topology feature vectorization information from the bird's-eye view feature map learned by the encoder. The road geometric features are represented as line segments consisting of the center line and its corresponding lane boundary. Each line segment is defined as an ordered set of N points in 3D space. Semantic features include lane classification (sidewalk, lane) and lane line type (dashed line, solid line, virtual line).

[0182] Please refer to Figure 7 Taking the detection of road elements at an intersection as an example, the local crowdsourcing information and forward-looking semantic information of the intersection are input into the road element detection model based on the crowdsourcing map. The road element detection model outputs the road element vector point set as follows: Figure 7 As shown in the right view, in this view, the lane centerline and lane markings are generated at the intersection, and the gene lane centerline generates road topology information, so that the vehicle can obtain the lane direction of the intersection, which is convenient for the automatic driving control of the vehicle. Similarly, or, obtain the crowdsourcing map corresponding to the intersection (left view). The crowdsourcing map can only detect the lane lines and sidewalks before entering the intersection. The lane centerline and lane markings cannot be generated in the intersection, and there is no topological information of the lane centerline. The automatic driving vehicle can only be detected after entering the intersection, or other high-precision equipment is used for detection. Compared with crowdsourcing maps or other detection methods, the road element detection method in the embodiment of the present application has low equipment cost while ensuring detection accuracy.

[0183] The embodiment of the present application also provides a training device for a road element detection model, the training device comprising:

[0184] A first acquisition module is used to acquire training samples, where the training samples include local crowdsourcing information corresponding to the target location acquired based on the crowdsourcing map and forward-looking semantic information corresponding to the target location;

[0185] A detection model is used to learn and obtain a road element vector point set at a target location based on input local crowdsourcing information and forward-looking semantic information, wherein the road element vector point set represents vectorized road elements and topological information of the road elements;

[0186] The training module is used to update the model parameters of the detection model according to the road element vector point set to obtain the road element detection model.

[0187] As an example, the training device also includes a data parsing module, which is used to parse the data sent by the DAT package under the Capilot framework. After positioning by ins_parser, the point cloud data corresponding to the multi-frame spliced ​​forward-looking map is projected to the self-vehicle bev perspective according to the set range, forming the forward-looking semantic information under the self-vehicle bev perspective map (30m*30m), representing the current environmental information of the target location; after positioning by ins_parser, the point cloud data corresponding to the local crowdsourcing data (or crowdsourcing map) is projected to the self-vehicle bev perspective according to the set range, forming the local crowdsourcing information under the self-vehicle bev perspective map (50m*100m), representing the historical crowdsourcing information corresponding to the target location. The forward-looking map and the crowdsourcing map are processed by one thread to form the forward-looking semantic information and the local crowdsourcing information, which are then put into the safety queue, and then obtained by another thread from the safety queue and passed into the road element detection model for reasoning.

[0188] The training device for the road element detection model provided in this embodiment can execute the training method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0189] The embodiment of the present application further provides a road element detection device, which is applied to a vehicle, and in particular to an autonomous driving vehicle, and the detection device includes:

[0190] The second acquisition module is used to obtain local crowdsourced information of the road the vehicle is currently traveling on based on the crowdsourced map, and is also used to obtain forward-looking semantic information of the vehicle on the road it is currently traveling on;

[0191] The road element detection model is used to learn the road element vector point set of the current driving road based on the input local crowdsourcing information and forward-looking semantic information. The road element vector point set represents the vectorized road elements and the topological information of the road elements.

[0192] As an example, the training device also includes a data parsing module, which is used to parse the data sent by the DAT package under the capilot framework, and after positioning by ins_parser, the point cloud data corresponding to the multi-frame spliced ​​forward-looking map is projected to the self-vehicle bev perspective according to the set range, forming the forward-looking semantic information under the self-vehicle bev perspective map (30m*30m), representing the real-time environmental information received during driving; after positioning by ins_parser, the point cloud data corresponding to the local crowdsourcing data (or crowdsourcing map) is projected to the self-vehicle bev perspective according to the set range, forming the local crowdsourcing information under the self-vehicle bev perspective map (50m*100m), representing the historical information of the current driving environment. The forward-looking map and the crowdsourcing map are processed by one thread to form the forward-looking semantic information and the local crowdsourcing information, which are then put into the safety queue, and then obtained by another thread from the safety queue and passed into the road element detection model for reasoning.

[0193] The detection device of the road element detection model provided in this embodiment can execute the detection method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0194] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor;

[0195] The memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions stored in the memory to implement the methods involved in the above embodiments.

[0196] The electronic device also includes a communication interface and a CAN bus, wherein the processor is used to provide computing power and control capabilities, and can be a GPU, CPU, NPU, MCU, FPGA, etc. The storage device includes an internal memory and a non-volatile memory. The non-volatile memory stores a computer program that implements the above method. The internal memory provides an environment for program startup and operation. The communication interface is used to communicate with an external terminal by wire or wireless.

[0197] In this embodiment, the electronic device may be an electronic component in a car, or may be the car.

[0198] The present invention also provides a computer-readable storage medium / computer program product, in which computer-readable storage medium stores computer execution instructions / the computer program product includes computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the methods involved in the above-mentioned embodiments.

[0199] As an example, the program consists of two parts: a data parsing module and a road element detection model. The data parsing module is used to parse the data sent by the DAT package under the capilot framework. After positioning by ins_parser, the point cloud data corresponding to the multi-frame spliced ​​forward-looking map is projected to the self-vehicle bev perspective according to the set range, forming the forward-looking semantic information under the self-vehicle bev perspective map (30m*30m), representing the real-time environmental information received during driving; after positioning by ins_parser, the point cloud data corresponding to the local crowdsourcing data (or crowdsourcing map) is projected to the self-vehicle bev perspective according to the set range, forming the local crowdsourcing information under the self-vehicle bev perspective map (50m*100m), representing the historical information of the current driving environment. The forward-looking map and the crowdsourcing map are processed by one thread to form the forward-looking semantic information and the local crowdsourcing information, which are then put into the security queue, and then obtained by another thread from the security queue and passed into the road element detection model for reasoning.

[0200] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A training method for a road element detection model, It is characterized in that The training method comprises: Acquire a training sample, wherein the training sample includes local crowdsourcing information corresponding to a target location acquired based on a crowdsourcing map and forward-looking semantic information corresponding to the target location; Inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model, learning and obtaining a road element vector point set at the target location, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements; The model parameters of the road element detection model are updated according to the road element vector point set.

2. The training method according to claim 1, It is characterized in that The step of inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model to learn and obtain a road element vector point set at the target location includes: Inputting the local crowdsourcing information and the forward-looking semantic information into the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information and extracting semantic map features based on the forward-looking semantic information in a feature extractor of the road element detection model, wherein the crowdsourcing map features and the semantic map features reflect road element features of different sizes and / or depths; Inputting the crowdsourced map features and the semantic map features into an encoder of the road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on a pre-learned bird's-eye view feature map, and updating the bird's-eye view feature map according to the learning results; The updated bird's-eye view feature map is input into the decoder of the road element detection model, and the geometric features, semantic features and road topology information of the road elements at the target location are extracted in the decoder based on the updated bird's-eye view feature map, and the road element vector point set at the target location is output based on the geometric features, the semantic features and the road topology information.

3. The training method according to claim 2, It is characterized in that The method of performing attention learning on the crowdsourcing map features and the semantic map features based on the pre-learned bird's-eye view feature map and updating the bird's-eye view feature map according to the learning result includes: Based on the bird's-eye view feature map obtained by pre-learning, spatial cross-attention interactive learning is performed on the crowdsourcing map features and the semantic map features to obtain cross-attention weights of the crowdsourcing map features and the semantic map features, and the learning result includes the cross-attention weights; The bird's-eye view feature map is updated according to the cross attention weights.

4. The training method according to claim 3, It is characterized in that The performing cross-attention interactive learning of the spatial positions of the crowdsourcing map features and the semantic map features to obtain the cross-attention weights of the crowdsourcing map features and the semantic map features includes: Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features; The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

5. The training method according to any one of claims 2 to 4, It is characterized in that The step of inputting the crowdsourced map features and the semantic map features into the encoder of the road element detection model, and performing attention learning on the crowdsourced map features and the semantic map features based on the pre-learned bird's-eye view feature map, comprises: Inputting the crowdsourced map features and the semantic map features into the encoder of the road element detection model, encoding the crowdsourced map features and the semantic map features, and aligning them in spatial position; Based on the pre-learned bird's-eye view feature map, attention learning is performed on the crowdsourced map features and semantic map features after encoding alignment.

6. The training method according to any one of claims 1 to 4, It is characterized in that The updating of the model parameters of the road element detection model according to the road element vector point set includes: Obtaining a loss value between an actual road element vector point set at the target location and the road element vector point set in a high-precision map; Model parameters of the road element detection model are updated based on the loss value.

7. A method for detecting road elements, It is characterized in that The detection method comprises: In autonomous driving, local crowdsourced information of a current road traveled by a vehicle is obtained based on a crowdsourced map, and forward-looking semantic information of the vehicle on the current road traveled is obtained; The local crowdsourcing information and the forward-looking semantic information are input into a road element detection model to obtain a road element vector point set of the current driving road, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements, wherein the road element detection model is obtained by the training method described in any one of claims 1 to 6.

8. The detection method according to claim 7, It is characterized in that The step of inputting the local crowdsourcing information and the forward-looking semantic information into a road element detection model to obtain a road element vector point set of the current driving road includes: Inputting the local crowdsourcing information and the forward-looking semantic information into the road element detection model, extracting crowdsourcing map features based on the local crowdsourcing information and extracting semantic map features based on the forward-looking semantic information in a feature extractor of the road element detection model, wherein the crowdsourcing map features and the semantic map features reflect road element features of different sizes and / or depths; Inputting the crowdsourced map features and the semantic map features into an encoder of the road element detection model, performing attention learning on the crowdsourced map features and the semantic map features based on a bird's-eye view feature map in the road element detection model, and obtaining a target bird's-eye view feature map according to the learning results; The target bird's-eye view feature map is input into the decoder of the road element detection model, in which the geometric features, semantic features and road topology information of the road elements of the current driving road are extracted based on the target bird's-eye view, and the road element vector point set of the current driving road is output based on the geometric features, the semantic features and the road topology information.

9. The detection method according to claim 8, It is characterized in that The method of performing attention learning on the crowdsourced map features and the semantic map features based on the bird's-eye view feature map in the road element detection model, and obtaining a target bird's-eye view feature map according to the learning result, includes: Based on the bird's-eye view feature map in the road element detection model, spatial cross-attention interactive learning is performed on the crowd-sourced map features and the semantic map features to obtain cross-attention weights of the crowd-sourced map features and the semantic map features, wherein the learning result includes the cross-attention weights; A target bird's-eye view feature map is obtained according to the cross attention weights.

10. The detection method according to claim 9, It is characterized in that The performing spatial cross-attention interactive learning on the crowd-sourced map features and the semantic map features to obtain the cross-attention weights of the crowd-sourced map features and the semantic map features, wherein the learning results include the cross-attention weights, including: Performing self-attention interactive learning at the temporal level on at least two adjacent frames of crowdsourced map features to obtain fused crowdsourced map features, and performing self-attention interactive learning at the temporal level on at least two adjacent frames of semantic map features to obtain fused semantic map features; The cross-attention interactive learning of the spatial position is performed on the fused crowdsourcing map features and the fused semantic map features to obtain the cross-attention weights of the fused crowdsourcing map features and the fused semantic map features.

11. A training device for a road element detection model, It is characterized in that The training device comprises: A first acquisition module is used to acquire a training sample, wherein the training sample includes local crowdsourcing information corresponding to a target location acquired based on a crowdsourcing map and forward-looking semantic information corresponding to the target location; A detection model, used for learning and obtaining a road element vector point set at the target location according to the input of the local crowdsourcing information and the forward-looking semantic information, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements; The training module is used to update the model parameters of the detection model according to the road element vector point set to obtain a road element detection model.

12. A road element detection device, It is characterized in that The detection device comprises: A second acquisition module is used to acquire local crowdsourced information of the road on which the vehicle is currently traveling based on the crowdsourced map, and is also used to acquire forward-looking semantic information of the vehicle on the road on which the vehicle is currently traveling; A road element detection model is used to learn and obtain a road element vector point set of the current driving road based on the input of the local crowdsourcing information and the forward-looking semantic information, wherein the road element vector point set represents the vectorized road elements and the topological information of the road elements.

13. An electronic device, It is characterized in that The electronic device comprises: a memory and a processor; The memory is used to store computer programs / instructions; the processor is used to implement the training method as described in any one of claims 1 to 6, or implement the detection method as described in any one of claims 7 to 10 according to the computer programs / instructions stored in the memory.

14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, is used to implement the training method as described in any one of claims 1 to 6, or to implement the detection method as described in any one of claims 7 to 10.

15. A computer program product, It is characterized in that The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements the training method as described in any one of claims 1 to 6, or implements the detection method as described in any one of claims 7 to 10.

Images