Road continuous element detection method, medium, product, electronic device and vehicle

Abstract

The application relates to the field of electronic equipment, in particular to a road continuous element detection method, medium, product, electronic equipment and vehicle. The road continuous element detection method comprises the following steps: acquiring a BEV feature map of a scene where a vehicle is located, and determining K initial query vectors of the BEV feature map; inputting the BEV feature map and the K initial query vectors into an attention network to obtain a fusion vector, wherein the attention network comprises M attention layers, the number of query vectors input into an Nth attention layer is less than the number of query vectors input into an (N+1)th attention layer, the number of query vectors input into an Mth attention layer is K, and 2<=N<=M; and based on the fusion vector, a detection result of a road continuous element in the BEV feature map is determined.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a method, medium, product, electronic equipment, and vehicle for detecting continuous road elements. Background Technology

[0002] In the field of driver assistance systems, the detection of continuous road features is a crucial step for driver assistance models to perceive the vehicle's driving environment. In existing technologies, to improve the detection accuracy of continuous road features, it is typically necessary to use a road continuous feature detection model based on the Transformer architecture to detect continuous features in the target road.

[0003] In some application scenarios, road continuous feature detection models need to run on automotive-grade edge devices. Due to the high time complexity of road continuous feature detection models and the limited computing power of automotive-grade edge devices, these devices are often unable to complete the detection process of continuous features in the target road within the target time required for actual driver assistance.

[0004] Therefore, to ensure the proper functioning of assisted driving models on automotive-grade edge devices, it is necessary to lightweight these models to enable smooth operation under limited computational resources. Current lightweighting techniques typically achieve this by compressing the network structure (e.g., pruning). However, another bottleneck in reducing computational resource consumption in the Transformer architecture lies in its attention module, and the aforementioned lightweighting methods are not very effective in reducing its computational resource consumption. Therefore, a lightweighting method targeting the attention mechanism is needed to alleviate the computational burden on edge devices running assisted driving models. Summary of the Invention

[0005] This application provides a method, medium, product, electronic device, and vehicle for detecting continuous road elements. The following describes this application from multiple aspects, and the embodiments and beneficial effects of these aspects can be referenced each other.

[0006] Firstly, this application provides a method for detecting continuous road features. The method includes: acquiring a BEV feature map of the scene where the vehicle is located, and determining K initial query vectors of the BEV feature map, where K is a positive integer greater than or equal to 2; inputting the BEV feature map and the K initial query vectors into an attention network to obtain a fusion vector, wherein: the attention network includes M attention layers, the number of query vectors input to the Nth attention layer is less than the number of query vectors input to the (N+1)th attention layer, and the number of query vectors input to the Mth attention layer is K, where 2 ≤ N. M; Based on the fusion vector, the detection results of continuous road elements in the BEV feature map are determined.

[0007] According to an embodiment of this application, a method for detecting continuous road features is provided. In this method, the number of query vectors used for calculation in each layer of the attention network is reduced and then increased during the attention calculation process, thereby reducing the time complexity of attention calculation and avoiding the problem that the problem of not being able to run on the edge with weak computing power due to too many query vectors is avoided.

[0008] In some embodiments, a BEV feature map of the scene where the vehicle is located is obtained, K feature points are selected on the BEV feature map to fit continuous road elements, and the BEV features of the K feature points are used as K initial query vectors.

[0009] In some embodiments, the input vector and BEV feature map of the i-th attention layer are input into the i-th attention layer to obtain the output vector of the i-th attention layer, where i is an integer sequentially selected from 1 to M, and:

[0010] When i equals 1, the K initial query vectors are used as the input vectors of the i-th attention layer;

[0011] When i is 2 to N, the output vector of the (i-1)th attention layer is aggregated, and the aggregated output vector is determined as the input vector of the i-th attention layer.

[0012] When i is N+1 to M, the output vector of the (i-1)th attention layer is diffused, and the diffused output vector is determined as the input vector of the i-th attention layer.

[0013] According to the embodiments of this application, the M-layer attention network is divided into two parts. In the first N layers, the number of input vectors to each layer is reduced, and in the last MN layers, the number of input vectors to each layer is increased, so as to reduce the amount of computation in each layer.

[0014] In some embodiments, the output vector of the (i-1)th attention layer is divided into multiple vector groups, and the number of output vectors in at least one vector group is two or more; the output vectors in each vector group are averaged to obtain the average vector corresponding to each vector group; the average vector corresponding to the multiple vector groups is used as the input vector of the i-th attention layer.

[0015] In some embodiments, the aggregation coefficient corresponding to the i-th attention layer is obtained. The aggregation coefficient is used to represent the ratio between the number of output vectors of the (i-1)-th attention layer and the number of input vectors of the i-th attention layer. Based on the aggregation coefficient corresponding to the i-th attention layer, the output vectors of the (i-1)-th attention layer are divided into multiple vector groups, wherein the number of output vectors in each vector group is the aggregation coefficient.

[0016] According to the embodiments of this application, the aggregation coefficients of each layer are used to determine how to aggregate the output vector of the previous attention layer in order to reduce the input vectors of each layer.

[0017] In some embodiments, the aggregation coefficients corresponding to each attention layer decrease sequentially from the second attention layer to the Nth attention layer.

[0018] In some embodiments, at least one output vector in the output vector of the (i-1)th attention layer is copied to obtain the copy vector corresponding to the (i-1)th attention layer; the output vector of the (i-1)th attention layer and the copy vector are used together as the input vector of the i-th attention layer.

[0019] In some embodiments, the diffusion coefficient corresponding to the i-th attention layer is obtained. The diffusion coefficient is used to represent the ratio between the number of input vectors of the i-th attention layer and the number of output vectors of the (i-1)-th attention layer. The diffusion coefficient is used as the replication factor to replicate the output vector of the (i-1)-th attention layer to obtain the replicated vector corresponding to the (i-1)-th attention layer.

[0020] According to the embodiments of this application, the diffusion coefficient of each layer is used to determine how to diffuse the output vector of the previous attention layer in order to increase the input vector of each layer.

[0021] In some embodiments, the diffusion coefficients corresponding to each attention layer increase sequentially from the Nth attention layer to the Mth attention layer.

[0022] In some embodiments, the BEV feature map is subjected to feature dimensionality reduction to obtain a dimensionality-reduced BEV feature map; the dimensionality-reduced BEV feature map is subjected to grid sampling to obtain a sampled feature map; the sampled feature map is subjected to feature dimensionality increase to obtain a dimensionality-increased feature map, the feature dimension of the dimensionality-increased feature map being the same as the feature dimension of the BEV feature map; attention operation is performed on the input vector of the i-th attention layer and the dimensionality-increased feature map to obtain the output vector of the i-th attention layer.

[0023] According to the embodiments of this application, in each attention layer, by first reducing the dimensionality of features and then increasing the dimensionality of features, the number of dimensions of the BEV feature map that needs to be processed by grid sampling is reduced. Since the time consumed by the feature reduction and feature increase methods is less than the time saved by grid sampling of the dimensionality-reduced BEV feature map, the time complexity of attention operations can be effectively reduced.

[0024] Secondly, embodiments of this application provide a readable storage medium including one or more programs, which, when executed on an electronic device, cause the electronic device to implement the method of any embodiment of the first aspect.

[0025] Thirdly, this application provides a program product that, when executed on an electronic device, causes the electronic device to implement the method of any embodiment of the first aspect.

[0026] Fourthly, this application provides an electronic device, comprising: at least one memory for storing one or more programs; and at least one processor for executing one or more programs to cause the electronic device to implement the method of any embodiment in the first aspect.

[0027] Fifthly, this application provides a vehicle that includes an electronic device as described in the fourth aspect. Attached Figure Description

[0028] Figure 1 A schematic diagram of the target scenario provided in the embodiments of this application;

[0029] Figure 2 A schematic diagram of a road element detection network provided in an embodiment of this application;

[0030] Figure 3 A flowchart illustrating the road continuity elements in the prior art, provided as an embodiment of this application;

[0031] Figure 4 A flowchart illustrating the road continuity elements provided in an embodiment of this application;

[0032] Figure 5 A flowchart illustrating the attention calculation method provided in an embodiment of this application;

[0033] Figure 6 A block diagram of an electronic device provided in an embodiment of this application;

[0034] Figure 7 This is a block diagram of a system-on-a-chip provided in an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application include, but are not limited to, a method, medium, product, electronic device, and vehicle for detecting continuous road features. The method provided by the embodiments of this application can realize the detection process of continuous features in a road.

[0036] For ease of understanding, this application first explains the terms that may be involved in the embodiments.

[0037] 1. Road Elements: In some embodiments, road elements may include various constituent objects that make up a road, wherein, based on their shape, road elements may be divided into continuous road elements and discrete road elements. For example, continuous road elements may typically be used to represent curved or polygonal objects (e.g., lane lines or center lines) extending along the road direction on a target road. Discrete road elements may represent objects that exist independently on the road (e.g., traffic lights or cones).

[0038] 2. BEV (Bird's-Eye View) Feature Map: In some embodiments, a BEV feature map is a feature map used to represent the top-view features of a target scene. For example, multi-view images of the target scene can be fused and transformed into a BEV space represented by a top-view perspective, and then features can be extracted from the transformed image to obtain the BEV feature map.

[0039] 3. Lightweighting: In some embodiments, lightweighting is a method of optimizing the model structure or reducing the number of parameters. Lightweighting methods enable deep learning models to run smoothly on resource-constrained devices (such as in-vehicle computing platforms) without reducing or only slightly reducing accuracy. For example, lightweighting of BEV perception models can be achieved through pruning, quantization, knowledge distillation, or designing efficient network structures, thereby improving the model's execution speed.

[0040] The following describes the overall process of the road element detection method in this application.

[0041] In some embodiments, the target scene to be detected can be provided by Figure 1 As shown, specifically, the target scenario can include a scenario where a vehicle is driving on a road.

[0042] For example, in Figure 1 In the middle, vehicle 110 is traveling on road 100, and cones 120 are placed on road 100 in front of vehicle 110 in the direction of travel. Road 100 is a one-way single-lane highway, and left lane line 101, right lane line 102 and center line 103 are marked on road 100.

[0043] Understandable, Figure 1In the target scene shown, since the cones 120 are placed independently, they can be considered as discrete road elements to be detected in the target scene. Furthermore, since the left lane line 101, right lane line 102, and center line 103 extend linearly and continuously, they can be considered as continuous road elements to be detected in the target scene.

[0044] In some embodiments, in order to detect the road elements including the cone 120, left lane line 101, right lane line 102, or center line 103, it can be done by... Figure 2 The road element detection model shown detects the aforementioned road elements.

[0045] Specifically, it can be like Figure 1 The image of the target scene shown is input into the road feature detection model to determine whether the image of the target scene contains continuous or discrete road features.

[0046] Specifically, Figure 2 According to some embodiments of this application, a schematic diagram of the structure of a road element detection module is shown. Figure 2 As shown, the road feature detection model 200 may include an image feature extraction module 210, a BEV deep spatiotemporal feature generation module 220, a multi-task detection head 230, a continuous road feature detection module 240, and a discrete road feature detection module 250.

[0047] The image feature extraction module 210 can extract shallow features of the target scene image to obtain a shallow feature map 260 of the target scene image.

[0048] The BEV deep spatiotemporal feature generation module 220 can further extract the BEV deep spatiotemporal features from the shallow feature map 260 to obtain the BEV feature map 270.

[0049] The multi-task detection head 230 can transmit the BEV feature map 270 to the road feature detection module corresponding to the type of the road feature to be detected (e.g., continuous road feature or discrete road feature) according to the type of the road feature to be detected (e.g., continuous road feature or discrete road feature).

[0050] The road continuous feature detection module 240 is used to perform target detection on road continuous features based on the features of road continuous features in the BEV feature map 270. For example, the road continuous feature detection module 240 can label the pixels corresponding to the center line 103 of the target scene as the center line based on the BEV feature map 270.

[0051] The road discrete feature detection module 250 is used to perform target detection on the road discrete features based on the features of the road discrete features in the BEV feature map 270. For example, the road discrete feature detection module 250 semantically labels the pixels corresponding to the cone 120 in the image of the target scene as cones.

[0052] In some embodiments, during the detection process of the road continuous feature detection module 240 as described above, the road continuous features can be represented by multiple discrete feature points on the BEV feature map 270.

[0053] It can be understood that when road continuous elements are represented by multiple feature points, the detection process of road continuous elements is essentially a process of fitting the road continuous elements to multiple feature points. When the overall feature pattern of multiple feature points matches the feature pattern corresponding to the road continuous element, the road continuous element fitted by these feature points can be identified as the detected road continuous element.

[0054] In some embodiments, each feature point used to fit the continuous road features as described above can be represented by a query vector, and then the continuous road features in the BEV feature map 270 can be fitted by the continuous road feature detection module 240 based on the attention fusion mechanism to realize the continuous road feature detection process.

[0055] Below, this application will use road continuity elements as an example, combined with Figure 3 The flowchart shown illustrates the specific process of detecting continuous road elements based on BEV feature map 270 in some current technical solutions.

[0056] Specifically, such as Figure 3 As shown, the specific process for detecting continuous road features based on BEV feature map 270 includes:

[0057] S300: Obtain the BEV feature map of the scene where the vehicle is located.

[0058] In some embodiments, the vehicle's location can be a real-world scenario where the vehicle 110 is actually driving. The vehicle can be a car, truck, bus, or other vehicles with different functions or shapes.

[0059] In some embodiments, the vehicle's location may include a road, which may include various different road types, such as urban roads, highways, tunnels, or bridges, etc., which are not limited herein.

[0060] It is understandable that, in multi-view images of the target scene, road elements may be segmented into multiple discontinuous segments due to scale changes, occlusion, and other reasons, which may lead to incomplete or discontinuous features of the road elements to be detected included in the feature map.

[0061] In some embodiments, to solve the above problems, a BEV feature map of the scene where the vehicle is located can be obtained from a top-down perspective. Since the BEV feature map can be obtained by merging the feature maps corresponding to multiple viewpoints into a single feature map and converting it into a top-down perspective, the road elements in the BEV feature map are complete and continuous, which can avoid problems such as missed detection in subsequent detection processes.

[0062] In some embodiments, the BEV feature map of the vehicle's location scene can be obtained through... Figure 2 The deep spatiotemporal feature generation module 220 of BEV is determined, and the specific method of determining the BEV feature map will not be elaborated here.

[0063] S310: Initialize K feature points and determine K initial query vectors corresponding to the K feature points.

[0064] It is understood that in some embodiments, the BEV feature map includes multiple pixels, where each pixel can be used to represent the BEV feature corresponding to a local region in the target scene image.

[0065] It is understandable that the length of continuous road features exceeds the range of local regions in the target scene image corresponding to each pixel on the BEV feature map. Therefore, the BEV features of continuous road features need to be represented by the BEV features of multiple pixels.

[0066] In some embodiments, in order to determine which pixel features can be used to further represent road continuity features, K feature points for fitting road continuity features can be initialized in the BEV feature map, where K is a positive integer greater than 2.

[0067] It is understandable that during the initialization of the initial feature points as described above, since the model has not yet learned which pixels in the BEV feature map can be used to represent continuous road features in the initial state, the K feature points can be obtained by randomly selecting K pixels. For example, the initial coordinates of the K feature points in the BEV feature map can be determined using a random seed algorithm.

[0068] In other embodiments, the feature point initialization method further includes uniform initialization. For example, K feature points with a preset distance between each pair of points in the BEV feature map can be determined by using a preset distance as the interval.

[0069] As can be understood, in the following text, a feature fusion method based on an attention mechanism will be used to fit continuous lane features using K initial feature points. Since the object operated on by the attention mechanism is the query vector, the query vector corresponding to the K feature points can be determined based on the features corresponding to the K feature points.

[0070] For example, if a feature point has T dimensions, the features of the T dimensions of the feature point can be defined as a query vector of length T, where T is a positive integer, such as 2, 3, 5, 10, 20 or 50, which is not limited in this application.

[0071] S320: Input the initial query vector of K feature points and the BEV feature map into the attention network.

[0072] In some embodiments, attention computation can be performed by an attention network, which may include a self-attention network, a cross-attention network, or a deformable attention network, and is not limited thereto.

[0073] In some embodiments, the number of layers in the attention network is M, where M is a positive integer greater than 2, such as 5, 6, 7, 10, 15, 20, 30 or 50, etc., which will not be elaborated further in this application.

[0074] S330: Determine if the current layer number i of the attention network is 1.

[0075] If the judgment result in S330 is yes, then execute S340, indicating that the current layer is the first layer of the attention network, and the initial query vector can be used as the input vector of this attention layer; if the judgment result is no, then execute S340A, indicating that the i-th attention layer is not the first layer of the attention network, and the output vector of the (i-1)-th attention layer can be used as the input vector of each attention layer.

[0076] The input vector of the i-th attention layer is the query vector used for feature fusion at the i-th attention layer. The output vector of the i-th attention layer is the output vector obtained after feature fusion at the i-th attention layer.

[0077] S340: Use the initial query vector as the input vector for the i-th attention layer.

[0078] It is understood that the input vector is a vector used to input the i-th attention layer to perform attention operations. In some embodiments, the input vector can be used as the query vector in the attention operation. The determination method of the key vector and value vector in the attention mechanism will be introduced later, and this application does not limit it here.

[0079] It is understandable that when i=1, the i-th attention layer is the first layer of the attention network. Therefore, the initial query vector input to the attention network in S320 can be used as the input vector.

[0080] S340A: Determine the output vector of the (i-1)th attention layer as the input vector of the i-th attention layer.

[0081] It can be understood that the output vector of the (i-1)th attention layer is the result obtained by the (i-1)th attention layer performing attention calculation on the input vector of the (i-1)th attention layer and the BEV feature map.

[0082] It is understandable that, when i ≠ 1, the i-th attention layer can be any layer other than the first layer of the attention network. Therefore, the output vector of the (i-1)-th layer can be determined as the input vector of the i-th layer.

[0083] S350: Input the input vector and BEV feature map of the i-th attention layer into the i-th attention layer to obtain the output vector of the i-th attention layer.

[0084] In some embodiments, the i-th attention layer can perform attention feature fusion on the input vector. The feature fusion can include performing self-attention feature fusion on the input vector of the i-th attention layer to obtain the self-attention fusion vector of the i-th attention layer. After obtaining the self-attention fusion vector, cross-attention feature fusion can be performed on the self-attention fusion vector according to the self-attention fusion vector and the BEV feature map to obtain the cross-attention fusion vector, which is the output vector of the i-th attention layer.

[0085] S360: Determine whether the current layer i of the attention network is M.

[0086] If the judgment result in S360 is yes, then execute S380, indicating that the feature fusion of the last layer of the attention network has been completed and the first fusion vector of K feature points has been obtained; if the judgment result is no, then execute S370, indicating that the attention network has not yet run to the last layer, and the output vector obtained from the i-th attention layer can be output to the (i+1)-th layer to continue feature fusion.

[0087] S370: Output the output vector of the i-th attention layer to the (i+1)-th layer and increment the value of i by one.

[0088] S380: Obtain the first fusion vector of K feature points.

[0089] In some embodiments, the first fusion vector may be the output vector of the last layer of the attention network.

[0090] It is understandable that after the attention feature fusion is completed, the K feature points corresponding to the first fusion vector can achieve the fitting of continuous road elements in the BEV feature map.

[0091] S390: Determine the detection results of continuous road features based on the first fusion vector of K feature points.

[0092] In some embodiments, the detection result of the road continuous element corresponding to the first fusion vector can be determined based on the road continuous element fitted by the K feature points obtained in S380.

[0093] For example, the first fusion vector can be input into a classification network (e.g., a fully connected network) to determine the location and type of continuous road features, such as the left lane line on the left side of the road, the right lane line on the right side of the road, or the center line in the middle of the road.

[0094] It is understandable that in S340A, the i-th attention layer can obtain the output vector by performing attention-based feature fusion calculation on the input vector, where the time complexity of attention-based feature fusion is O(n). ), and n is a parameter related to the number of feature points.

[0095] It is understandable that for a time complexity of O( The time consumed by attention-based feature fusion algorithms increases exponentially with the number of feature points.

[0096] In some embodiments, the road continuous feature detection module 240 needs to run on an automotive-grade edge device (e.g., a vehicle-mounted system). It is understood that the computational resources of an automotive-grade edge device cannot handle a time complexity of O(n) when the number of feature points is too large. The computational task.

[0097] Therefore, in order to enable the aforementioned continuous road feature detection module to run on computationally limited devices such as automotive-grade edge devices, it is necessary to lightweight the module to allow it to operate on such devices. However, currently used lightweighting methods, such as pruning or low-rank approximation, can only sparsify or reduce the dimensionality of static weights in the network, and cannot lightweight the attention feature fusion algorithm from the perspective of feature point scale.

[0098] To this end, this application proposes a method for detecting continuous road features, comprising: initializing an initial query vector corresponding to K feature points, then inputting the initial query vector into an M-layer attention network; in the first N layers, merging the output vectors of the (i-1)th layer to obtain the input vector of the i-th layer and performing feature fusion on the input vector; in the last MN layers, expanding the output vector of the (i-1)th layer to obtain the input vector of the i-th layer and performing feature fusion on the input vector; and the number of vectors corresponding to the output vector of the M-th layer network is K.

[0099] Based on the detection method described above, in an M-layer attention network, the number of vectors corresponding to the input vectors of the first N layers is reduced, and then the number of vectors corresponding to the input vectors of the last MN layers is increased to N. This ensures that the number of vectors corresponding to the input vectors of each layer in the M-layer attention network is no greater than N. Therefore, compared to current technical solutions, the number of vectors used for attention calculation in each layer is reduced, achieving computational lightweighting.

[0100] For example, with N=8, M=4, and K=3, in some current technical solutions, the time complexity is O(n). The attention fusion algorithm of the previous method requires 8*8*4=256 computations. However, in the technical solution of this application, if the number of input vectors corresponding to the first to fourth layers are set to 8, 4, 2, 8 respectively, then the time complexity is O(...). The attention fusion algorithm of ) is calculated 8*8+4*4+2*2+8*8=148 times.

[0101] The following describes a method for detecting continuous road elements based on some embodiments of this application.

[0102] Specifically, Figure 4 A flowchart of a road continuity feature detection method provided in an embodiment of this application is shown. Figure 4 As shown, the methods for detecting continuous road features include:

[0103] S400: Obtain the BEV feature map of the scene where the vehicle is located.

[0104] S410: Initialize K feature points and determine K initial query vectors corresponding to the K feature points.

[0105] S420: Input the initial query vector of K feature points and the BEV feature map into the attention network.

[0106] It is understood that the initialization process of the initial query vector has not been improved in this application. Therefore, the specific process of S400 to S420 can be referred to S300 to S320, and will not be repeated here.

[0107] S430: Determine if the current layer number i of the attention network is 1.

[0108] If the judgment result in S430 is yes, then execute S440, indicating that the current layer is the first layer of the attention network, and the initial query vector of K feature points can be used as the input vector of the first layer network; if the judgment result is no, then execute S440A, indicating that the i-th layer network is not the first layer of the attention network, and the input vector of the i-th layer attention network can be determined according to the current layer number i.

[0109] In this context, the number of layers i in the attention network is a positive integer increasing sequentially from 1 to M, such as 1, 2, or 3.

[0110] It can be understood that attention networks fuse features from the query vectors input to each attention layer in ascending order of layer number. The query vector input to each attention layer can be determined as the input vector for that specific attention layer. Therefore, the input vector to each attention layer can be determined by its specific layer number. The specific determination method will be described later and will not be elaborated upon here.

[0111] S440: Use the initial query vector as the input vector for the i-th attention layer.

[0112] It can be understood that when i=1, the i-th attention layer is the first layer of the attention network. Since this layer is the first layer of the attention network, it needs to receive the initial query vector input to the attention network in S420. In other words, the above initial query vector is used as the input vector of the first attention layer.

[0113] S440A: Determine whether the current layer number i of the attention network is less than or equal to N.

[0114] If the judgment result in S440A is yes, then S440B is executed, indicating that the number of vectors corresponding to the feature vectors used to fit the continuous road features can be reduced layer by layer to reduce the computational time complexity; if the judgment result is no, then S440C is executed, indicating that the number of feature vectors used to fit the continuous road features can be increased layer by layer to ensure that the number of vectors corresponding to the final output vector of the model is still K.

[0115] Wherein, N is a positive integer greater than 1 and less than M, such as 2, 3 or M / 2, etc., which are not limited in this application.

[0116] In some embodiments, when i ≠ 1 and i is a positive integer taking values ​​sequentially from 2 to M, the input vector of the i-th attention layer can be obtained by performing a first processing on the output vector of the (i-1)-th attention layer. The output vector of the (i-1)-th attention layer may include a vector obtained by fusing attention features with the input vector of the (i-1)-th attention layer.

[0117] It is understandable that, in order to lightweight the attention network, the specific operations of the first processing described above can be determined based on the specific value of the layer number i.

[0118] In some embodiments, when i is less than or equal to N, the first process described above may include an aggregation process.

[0119] As can be understood, aggregation processing is a method of aggregating at least two vectors from the output vectors of the (i-1)th attention layer. This method ensures that the number of input vectors for the i-th layer after aggregation is less than the number of output vectors for the (i-1)th attention layer, thus reducing the number of input vectors for the i-th attention layer.

[0120] It is understandable that as the number of attention network layers increases, the input vectors to each attention layer decrease layer by layer due to the aggregation process. This continues until the input vector of the i-th attention layer reaches its minimum when i=N.

[0121] The specific execution method of the aggregation process will be described in S440B, and will not be repeated here.

[0122] When i is greater than N, as i takes values ​​from N+1 to M in sequence, the first operation described above may include diffusion processing.

[0123] It can be understood that diffusion processing is a method of spreading at least one vector in the output vector of the (i-1)th attention layer. This method ensures that the number of input vectors for the i-th layer after diffusion is greater than the number of output vectors for the (i-1)th attention layer, thereby increasing the number of input vectors for the i-th attention layer.

[0124] It is understandable that as the number of attention network layers increases, the input vectors to each attention layer increase layer by layer due to the diffusion process. Furthermore, when i=M, the number of input vectors to the i-th attention layer is N, thus ensuring that the number of initial query vectors input to the entire attention network is equal to the number of fused vectors output.

[0125] The specific implementation method of diffusion treatment will be described in S440C, and will not be repeated here.

[0126] It's understandable that without lightweighting the attention network, with 4 layers and 16 computations per layer, the total computation count would be 64. However, by lightweighting the attention network, the computation count for each layer is less than or equal to 16, resulting in a total computation count less than 64, thus achieving lightweighting.

[0127] S440B: Merge the output vectors of the (i-1)th attention layer and determine the merged output vector as the input vector of the i-th layer.

[0128] It is understandable that, as described in S440A, when i is less than or equal to N, in order to reduce the number of input vectors of the i-th attention layer, the output vectors of the (i-1)-th layer can be aggregated.

[0129] In some embodiments, the aggregation process can be implemented by grouping the output vectors of the (i-1)th layer and averaging the vectors within each group.

[0130] Specifically, in order to determine which output vectors to average, the output vectors of the (i-1)th attention layer can be divided into multiple vector groups.

[0131] In some embodiments, multiple vector groups can be divided by an aggregation coefficient, which represents the ratio between the number of output vectors of the (i-1)th attention layer and the number of input vectors of the ith attention layer.

[0132] Therefore, in order to divide the vector groups, the aggregation coefficients corresponding to the i-th attention layer can be obtained first, and the output vectors of the (i-1)-th attention layer can be divided into multiple vector groups according to the aggregation coefficients corresponding to the i-th attention layer. The number of output vectors in each vector group is the aggregation coefficient.

[0133] For example, when the aggregation coefficient is 2, the output vector of the (i-1)th attention layer can be divided into multiple vector groups, each containing two output vectors.

[0134] It is understood that the aggregation coefficients corresponding to each layer can be positive integers that decrease sequentially, such as 4, 3 or 2, and this application does not impose any restrictions on them.

[0135] After obtaining the divided vector groups, the output vectors in each vector group can be averaged to obtain the average vector corresponding to each vector group.

[0136] In some embodiments, the averaging described above includes averaging the individual feature dimensions of the output vectors in each vector group. For example, if a vector group has two output vectors and each output vector has eight feature dimensions, the eight feature dimensions of the two output vectors can be averaged separately to obtain an average vector corresponding to the vector group.

[0137] After averaging all vector groups, the average vector corresponding to all vector groups can be used as the input vector of the i-th attention layer.

[0138] For example, if there are 8 output vectors in the (i-1)th attention layer, the vectors in the 8 output vectors of the (i-1)th attention layer can be randomly divided into 4 groups, with each group including two output vectors. The average of the two output vectors in each group is then taken, and the 4 new vectors obtained after averaging are used as the input vectors of the i-th attention layer.

[0139] S440C: Expand the output vector of the (i-1)th attention layer and use the expanded output vector as the input vector of the i-th layer.

[0140] It is understandable that, as described in S440A, when i is greater than N, in order to increase the number of input vectors of the i-th attention layer, the output vector of the (i-1)-th layer can be diffused.

[0141] In some embodiments, the diffusion process can be implemented by copying at least one of the multiple output vectors of the (i-1)th layer.

[0142] In some embodiments, the number of times at least one output vector is copied in the diffusion process can be determined by a diffusion coefficient, which represents the ratio between the number of input vectors of the i-th attention layer and the number of output vectors of the (i-1)-th attention layer.

[0143] Therefore, in order to copy at least one of the multiple output vectors of the (i-1)th layer, the diffusion coefficient corresponding to the i-th attention layer can be obtained first, and the diffusion coefficient corresponding to the i-th attention layer can be used as the copy to copy the output vector of the (i-1)th attention layer.

[0144] For example, when the diffusion coefficient is 2, each vector in the output vector of the (i-1)th attention layer can be copied once to obtain a copy vector with the same number of copies as the output vector of the (i-1)th attention layer.

[0145] It is understood that the diffusion coefficients corresponding to each layer can be positive integers that increase sequentially, such as 1, 2 or 3, and this application does not impose any restrictions on them.

[0146] It is understandable that, in some embodiments, to ensure consistency in the network structure stacking, the output vector of the last layer of the attention network needs to be consistent with the initial query vector. Therefore, in an M-layer attention network, the number of input vectors for the M-th attention layer needs to be set to K. This ensures that the number of output vectors for the M-th attention layer is K, consistent with the number of vectors in the initial query vector.

[0147] S450: Input the input vector and BEV feature map of the i-th attention layer into the i-th attention layer to obtain the output vector of the i-th attention layer.

[0148] S460: Determine if the current layer i of the attention network is M.

[0149] S470: Output the output vector of the i-th attention layer to the (i+1)-th layer and increment the value of i by one.

[0150] S480: Obtain the first fusion vector of K feature points.

[0151] S490: Determine the detection results of continuous road features based on the first fusion vector of K feature points.

[0152] It is understood that S450 to S490 are used to determine whether the network has reached the last layer and obtain the detection results of the continuous road elements based on the output vector of the last layer. The specific process of S450 to S490 can be referred to the specific process of S350 to S390, which will not be repeated here.

[0153] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any storage medium (e.g., magnetic medium, optical medium, semiconductor medium, etc.) capable of storing and / or retrieving data by a computing device. The computer-readable storage medium includes instructions that direct a computing device to implement the road continuity feature detection method provided in this application.

[0154] This application also provides a program product, which may be a software or program product including instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, the at least one computing device implements the road continuity feature detection method provided in this application.

[0155] This application also provides an electronic device, including one or more processors and one or more memories, wherein the one or more processors can be used to execute instructions to implement the road continuous feature detection method provided in this application.

[0156] This application also provides a vehicle. The vehicle includes one or more electronic devices that can be used to perform the road continuity feature detection method provided in this application.

[0157] Figure 5 The flowchart illustrating the attention calculation process performed by each attention layer according to an embodiment of this application is shown, such as... Figure 5 As shown, it includes:

[0158] S510: Perform feature dimensionality reduction on the BEV feature map to obtain a dimensionality-reduced BEV feature map.

[0159] In some embodiments, the BEV feature map is also mesh-sampled before attention computation is performed to achieve spatial alignment and normalization of the features.

[0160] It is understandable that the mesh sampling performed in S520 operates on each dimension of the feature corresponding to each pixel in the BEV feature map, which leads to high computational complexity and increases the computational latency and computational resource consumption of the edge device.

[0161] Therefore, in some embodiments, feature dimensionality reduction is performed on the BEV feature map before mesh sampling to reduce the computational cost of the mesh sampling operation.

[0162] For example, a high-dimensional feature channel can be compressed to a lower dimension using a 1*1 convolution kernel to obtain a dimension-reduced BEV feature map, which is not limited in this application.

[0163] S520: Perform grid sampling on the dimensionality-reduced BEV feature map to obtain the sampled feature map.

[0164] It is understandable that grid sampling can map the irregularly distributed feature points in the BEV feature map onto a regular grid to obtain a sampled feature map.

[0165] It is understandable that while feature dimensionality reduction in S510 and feature dimensionality increase in S530 also require computational resources and time, feature dimensionality reduction reduces the feature dimension of the BEV feature map. Therefore, in the calculation process of S520, more computational resources or time can be saved, thus achieving the goal of saving computational resources or reducing computational complexity overall.

[0166] S530: Perform feature upscaling on the sampled feature map to obtain an upscaled feature map.

[0167] It is understandable that since the feature map used for grid sampling in S520 is a dimensionality-reduced BEV feature map, the feature dimension of the obtained sampled feature map is also dimensionality-reduced. Therefore, in order to perform attention calculations later, the sampled feature map obtained above needs to be dimensionality-upgraded to obtain the dimensionality-upgraded feature map.

[0168] For example, the dimension of the feature can be increased by performing deconvolution operation using a 1*1 convolution kernel to obtain the same feature dimension as the BEV feature map. This application does not limit this.

[0169] S540: Perform attention operations on the input vector and the upgraded feature map of the i-th attention layer to obtain the output vector of the i-th attention layer.

[0170] In some embodiments, for ease of representation, the upgraded feature map can also be represented in the form of a vector. Therefore, after obtaining the input vector and upgraded feature map of the i-th layer, attention calculation is performed to obtain the output vector of the i-th attention layer.

[0171] For example, the input vector can be used as the query vector in the attention calculation, and the upgraded feature map can be used as the key vector and value vector in the attention calculation. Attention calculation can be performed on the input vector to obtain the output vector.

[0172] It is understandable that attention calculation can be used to adaptively focus on the spatial location in the BEV feature map that is most relevant to the road continuous elements, so that the model can accurately determine the topological structure and geometric continuity of the road continuous elements in the BEV space.

[0173] For example, Figure 6 A block diagram of an electronic device 400 according to an embodiment of this application is shown. The electronic device 400 may include one or more first processors 401 coupled to a controller hub 403. In at least one embodiment, the controller hub 403 communicates with the first processors 401 via a multi-branch bus such as a Front Side Bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar network interface 406. The first processors 401 execute instructions controlling general types of data processing operations. In some embodiments, the controller hub 403 may include, but is not limited to, a Graphics & Memory Controller Hub (GMCH) (not shown) and an Input / Output Hub (IOH) (which may be on separate chips) (not shown), wherein the GMCH may include memory and a graphics controller and is coupled to the IOH.

[0174] Electronic device 400 may further include a first coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in the embodiments of this application), with memory 404 and the first coprocessor 402 directly coupled to the first processor 401 and the controller hub 403, which is located on a single chip with the IOH. Memory 404 may be, for example, Dynamic Random Access Memory (DRAM), Phase Change Memory (PCM), or a combination of both.

[0175] The memory 404 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of those instructions. The instructions may include, when executed by at least one of the processors, causing the electronic device 400 to perform, as... Figure 4 The instructions for the method are shown. When the instructions are executed on a computer, the computer performs the digital asset insertion method disclosed in the embodiments of this application.

[0176] In one embodiment, the first coprocessor 402 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on graphics processing units (GPGPU), or an embedded processor, etc. Optional properties of the first coprocessor 402 are indicated by dashed lines. Figure 6 middle.

[0177] In another embodiment, electronic device 400 may further include a Network Interface Controller (NIC) 406. Network interface 406 may include a transceiver for providing a radio interface for electronic device 400 to communicate with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, network interface 406 may be integrated with other components of electronic device 400. Network interface 406 can implement the functions of the communication unit in the above embodiments.

[0178] Electronic device 400 may further include input / output (I / O) devices 405. I / O 405 may include: a user interface designed to enable a user to interact with electronic device 400; a peripheral component interface designed to enable peripheral components to also interact with electronic device 400; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 400.

[0179] It is worth noting that, Figure 6 This is merely an example. That is, although... Figure 3 The electronic device 400 shown includes multiple devices such as a first processor 401, a controller hub 403, and a memory 404. However, in actual applications, devices using the methods of the embodiments of this application may include only a portion of the devices in the electronic device 400. For example, it may include only the first processor 401 and the network interface 406. Figure 6 The properties of the optional devices are shown by dashed lines.

[0180] Figure 7 A block diagram of a System-on-Chip (SoC) 500 applied to an electronic device according to an embodiment of this application is shown. Figure 7 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 7 In this embodiment, SoC 500 includes: an interconnect unit 550 coupled to a second processor 510; a system proxy unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more second coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random-access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the second coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC processor, or an embedded processor.

[0181] Static Random Access Memory (SRAM) cell 530 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of those instructions. These instructions may include: instructions that, when executed by at least one processor, cause the SoC to implement the methods shown in the embodiments of this application. When the instructions are executed on a computer, they cause the computer to perform the truth-matching method disclosed in the embodiments of this application.

[0182] It should be noted that the terminology used in the embodiment section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between associated obstacles, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. In addition, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more.

[0183] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0184] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0185] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product may include one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium, etc.

[0186] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium can include various media capable of storing program code, such as read-only memory, random access memory, magnetic disks, or optical disks.

[0187] The above description is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

1. A method for detecting continuous road features, characterized in that, The method includes: Obtain the BEV feature map of the scene where the vehicle is located, and determine K initial query vectors of the BEV feature map, where K is a positive integer greater than 2; The BEV feature map and K initial query vectors are input into an attention network to obtain a fused vector, wherein: the attention network includes M attention layers, the number of query vectors input to the Nth attention layer is less than the number of query vectors input to the (N+1)th attention layer, and the number of query vectors input to the Mth attention layer is K, 2≤N M; Based on the fusion vector, the detection results of road continuous elements in the BEV feature map are determined; The step of inputting the BEV feature map and K initial query vectors into the attention network to obtain the fusion vector includes: The input vector of the i-th attention layer and the BEV feature map are input into the i-th attention layer to obtain the output vector of the i-th attention layer, where i is an integer taking values ​​sequentially from 1 to M; where: When i equals 1, the K initial query vectors are used as the input vectors of the i-th attention layer; When i is 2 to N, the output vector of the (i-1)th attention layer is aggregated, and the aggregated output vector is determined as the input vector of the i-th attention layer. When i is N+1 to M, the output vector of the (i-1)th attention layer is diffused, and the diffused output vector is determined as the input vector of the i-th attention layer.

2. The method according to claim 1, characterized in that, The process of obtaining the BEV feature map of the vehicle's location scene and determining K initial query vectors for the BEV feature map includes: Obtain the BEV feature map of the scene where the vehicle is located, select K feature points on the BEV feature map to fit the continuous road elements, and use the BEV features of the K feature points as the K initial query vectors.

3. The method according to claim 1, characterized in that, When i is 2 to N, the process of aggregating the output vector of the (i-1)th attention layer and determining the aggregated output vector as the input vector of the i-th attention layer includes: The output vector of the (i-1)th attention layer is divided into multiple vector groups, and at least one of the vector groups contains two or more output vectors. The average of the output vectors in each vector group is obtained to obtain the average vector corresponding to each vector group. The average vector corresponding to the multiple vector groups is used as the input vector of the i-th attention layer.

4. The method according to claim 3, characterized in that, The step of dividing the output vector of the (i-1)th attention layer into multiple vector groups includes: Obtain the aggregation coefficient corresponding to the i-th attention layer, whereby the aggregation coefficient represents the ratio between the number of output vectors of the (i-1)-th attention layer and the number of input vectors of the i-th attention layer. Based on the aggregation coefficients corresponding to the i-th attention layer, the output vector of the (i-1)-th attention layer is divided into multiple vector groups, where the number of output vectors in each vector group is the aggregation coefficient.

5. The method according to claim 4, characterized in that, The aggregation coefficients corresponding to each attention layer decrease sequentially from the 2nd attention layer to the Nth attention layer.

6. The method according to claim 1, characterized in that, When i is N+1 to M, the process of performing diffusion processing on the output vector of the (i-1)th attention layer and determining the diffused output vector as the input vector of the i-th attention layer includes: Copy at least one of the output vectors of the (i-1)th attention layer to obtain the copy vector corresponding to the (i-1)th attention layer. The output vector and copy vector of the (i-1)th attention layer are used together as the input vector of the i-th attention layer.

7. The method according to claim 6, characterized in that, At least one output vector from the (i-1)th attention layer is copied to obtain a copy vector corresponding to the (i-1)th attention layer, including: Obtain the diffusion coefficient corresponding to the i-th attention layer, whereby the diffusion coefficient represents the ratio between the number of input vectors of the i-th attention layer and the number of output vectors of the (i-1)-th attention layer. Using the diffusion coefficient as the replication factor, the output vector of the (i-1)th attention layer is replicated to obtain the replicated vector corresponding to the (i-1)th attention layer.

8. The method according to claim 7, characterized in that, The diffusion coefficients corresponding to each attention layer increase sequentially from the Nth attention layer to the Mth attention layer.

9. The method according to claim 1, characterized in that, The step of inputting the input vector of the i-th attention layer and the BEV feature map into the i-th attention layer to obtain the output vector of the i-th attention layer includes: The BEV feature map is subjected to feature dimensionality reduction to obtain a dimensionality-reduced BEV feature map; The reduced-dimensional BEV feature map is subjected to grid sampling to obtain a sampled feature map; The sampled feature map is up-dimensional to obtain an up-dimensional feature map, and the feature dimension of the up-dimensional feature map is the same as the feature dimension of the BEV feature map. Attention operations are performed on the input vector of the i-th attention layer and the upgraded feature map to obtain the output vector of the i-th attention layer.

10. A readable storage medium, characterized in that, The readable storage medium includes one or more programs that, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 9.

11. A program product, characterized in that, When the program product is executed on an electronic device, it causes the electronic device to perform the method described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: At least one memory for storing one or more programs; At least one processor is configured to execute the one or more programs to cause the electronic device to perform the method of any one of claims 1 to 9.

13. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 12.

Images