Map element rapid attention calculation method and system, electronic equipment and medium

Abstract

The invention provides a map element rapid attention calculation method and system, electronic equipment and a computer readable storage medium, and the method comprises the steps: multiplying a point set query vector with linear output data to obtain point set attention output, and multiplying an element query vector with the linear output data to obtain element attention output, according to the method, the point set and the elements are decoupled, then the point set and the elements are added to obtain attention output, the decoupled point set and the elements are quickly coupled, the attention output is calculated based on linear output, attention calculation only needs to be conducted once, the calculation amount is not additionally increased while attention calculation is conducted, decoupling between the map element instances and the point set is achieved in the whole process, and the calculation efficiency is improved. Moreover, the calculation amount is small, the defect that the attention module needs to be executed for multiple times is avoided, the problem that the complexity is too high due to a long sequence is also avoided, the attention calculation effect is guaranteed, meanwhile, the efficiency is obviously improved, the operation complexity is reduced, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and more specifically to a method for calculating attention of map elements during autonomous driving, and more specifically to a method, system, electronic device and medium for quickly calculating attention of map elements. Background Art

[0002] In recent years, deep learning technology has been widely used in the field of autonomous driving, and transformers have become the basic architecture in this field. In the field of autonomous driving, the detection of map elements such as lane lines, roadside edges, and zebra crossings is a basic task. At present, one method in the prior art is to model map elements as a set of points with equivalent arrangements, and use a CNN combined with a transformer architecture to complete the map element detection task. Although this method has a good detection effect, the high computational cost of transformers makes it still challenging to deploy on devices with limited computing power.

[0003] For example, for the map element detection task, assume that there are N elements in the image, and each element consists of M points. If the transformer architecture is used, there will be N*M queries (query requests), and N*M is generally greater than 1000. If the traditional transformer structure is used, the sequence is long and the amount of calculation is huge.

[0004] In order to reduce the computational cost, some researchers decoupled the elements from the point set and calculated the attention separately, but the attention module needs to be executed twice, and this method is still inefficient; some researchers replaced the softmax function in the attention module with ReLU, which improved the computational efficiency to a certain extent, but when the sequence is too long, its computational efficiency is still not ideal. Some researchers also proposed a linear attention calculation method to further improve the computational efficiency of attention, but for map element detection based on the transformer architecture, the complexity is still high due to the long query sequence.

[0005] Therefore, there is an urgent need for a method, system, electronic device and medium for fast attention calculation of map elements that can avoid the shortcomings of multiple executions of the attention module, reduce computational complexity, reduce the amount of calculation, and improve the efficiency of attention calculation. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a method, system, electronic device and medium for quickly calculating the attention of map elements, so as to solve the technical problems in the prior art of calculating the attention of map elements and the attention of point sets, which are either calculated in a decoupled manner with large amount of calculation or calculated in a non-decoupled manner with high complexity.

[0007] In a first aspect, an embodiment of the present application provides a method for quickly calculating attention of a map element, and the method for quickly calculating attention of a map element includes:

[0008] Get the query vector, key vector and value vector of the map element input into the preset transformer model;

[0009] Performing dimension compression on the query vector to obtain a point set query vector and an element query vector, and performing linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0010] multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output;

[0011] The query vector is linearly mapped using grouped convolution to obtain mapping data, and the mapping data is added to the attention output to obtain an attention result.

[0012] Optionally, the step of obtaining a query vector, a key vector, and a value vector about a map element input into a preset transformer model includes:

[0013] Get query data, key data and numerical data about map elements input into the preset transformer model;

[0014] The query data, key data and numerical data are matrix-converted to obtain a query vector, a key vector and a numerical vector having four-dimensional matrix dimensions, respectively.

[0015] Optionally, the step of performing dimension compression on the query vector to obtain a point set query vector includes:

[0016] Obtaining the matrix dimension of the query vector;

[0017] Taking the mean over a second dimension of the matrix dimensions of the query vector to obtain a first mean;

[0018] The first mean is dimensionally transformed to obtain a point set query vector.

[0019] Optionally, the step of performing dimension compression on the query vector to obtain an element query vector includes:

[0020] Obtaining the matrix dimension of the query vector;

[0021] Taking the mean over a third dimension of the matrix dimensions of the query vector to obtain a second mean;

[0022] The second mean is dimensionally transformed to obtain an element query vector.

[0023] Optionally, the step of linearly mapping the key vector to obtain a key linear output includes:

[0024] Input the key vector into a preset first linear processing unit for convolution activation linear processing to obtain a key linear vector; wherein the first linear processing unit includes a preset convolution layer and an activation function layer, the convolution kernel of the convolution layer is 1x1xdx(n*dim), and the activation function layer adopts a relu activation function;

[0025] The key linear vector is deformed to obtain a key linear output.

[0026] Optionally, the step of performing linear mapping on the numerical vector to obtain a numerical linear output comprises:

[0027] Performing convolution linear processing on the numerical vector input value in a second linear processing unit preset by the second linear processing unit to obtain a numerical linear vector; wherein the second linear processing unit includes a convolution layer; and the convolution kernel of the convolution layer is 1x1xdx(n*dim);

[0028] The numerical linear vector is transformed to obtain a numerical linear output.

[0029] Optionally, the step of performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data includes:

[0030] multiplying the matrix of key linear outputs by the matrix of numerical linear outputs to obtain a linear product;

[0031] The linear product is divided by a preset control variance function to obtain linear output data.

[0032] In a second aspect, an embodiment of the present application provides a system for calculating the rapid attention of map elements, which implements the method for calculating the rapid attention of map elements as described above, and the system includes:

[0033] A dimension vector acquisition unit, used to acquire a query vector, a key vector and a value vector about a map element input into a preset transformer model;

[0034] A dimension compression unit, used for performing dimension compression on the query vector to obtain a point set query vector and an element query vector;

[0035] A linear mapping unit, used to perform linear mapping on the key vector and the numerical vector respectively to obtain a key linear output and a numerical linear output, and perform matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0036] a linear coupling unit, multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output;

[0037] A result output unit is used to linearly map the query vector using grouped convolution to obtain mapping data, and add the mapping data to the attention output to obtain an attention result.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, the one or more programs comprising instructions, which, when executed by the processor, enable the processor to execute any one of the optional map element rapid attention calculation methods described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for quickly calculating attention of map elements as optionally described in the first aspect is implemented.

[0040] It can be seen from the above technical solutions that the embodiments of the present application provide a method, system, electronic device and computer-readable storage medium for calculating the rapid attention of map elements. Compared with the prior art, the present application has the following beneficial effects:

[0041] First, the query vector is dimensionally compressed to obtain a point set query vector and an element query vector. In this step, the point set and the element are quickly decoupled without increasing the amount of calculation.

[0042] Then, linear mapping is performed on the key vector and the numerical vector respectively to obtain the key linear output and the numerical linear output, and matrix multiplication is performed based on the key linear output and the numerical linear output to obtain the linear output data, and then the point set query vector is multiplied by the linear output data to obtain the point set attention output, and the element query vector is multiplied by the linear output data to obtain the element attention output, and the two are added to obtain the attention output, so as to quickly couple the decoupled point set and element, and calculate the attention output based on the linear output, which only needs to calculate the attention once, and when calculating the attention, no additional calculation amount is added;

[0043] Finally, grouped convolution is used to perform linear mapping on the query vector to obtain mapping data, and the mapping data is added to the attention output to obtain the attention result. The whole process not only realizes the decoupling between the map element instance and the point set, but also has a small amount of calculation. It avoids the disadvantage of needing to execute the attention module multiple times and the problem of high complexity caused by long sequences. While ensuring the attention calculation effect, its efficiency is significantly improved, reducing the computational complexity and improving the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0045] Figure 1 is a flowchart of a method for quickly calculating attention of map elements according to an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the overall logic operation in the method for fast attention calculation of map elements according to an embodiment of the present invention;

[0047] Figure 3 A logic block diagram of a map element fast attention calculation system according to an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The existing methods for calculating the attention of map elements and the attention of point sets generally execute the attention module twice, respectively calculating the attention between elements and the attention between point sets. This method increases the amount of calculation; there are also methods that do not decouple elements and point sets. Although only one attention module needs to be executed, the query sequence is long and the complexity is high. That is, the existing technology either uses a decoupled method for calculation, which has a large amount of calculation, or uses a non-decoupled method for calculation, but the complexity is high.

[0050] In view of the above problems, the present invention provides a method, system, electronic device and computer-readable storage medium for fast attention calculation of map elements. First, a query vector, a key vector and a numerical vector about a map element input into a transformer model are obtained, and then the query vector is dimensionally compressed to obtain a point set query vector and an element query vector, and the key vector and the numerical vector are linearly mapped to obtain a key linear output and a numerical linear output respectively, and matrix multiplication is performed based on the key linear output and the numerical linear output to obtain linear output data, and then the point set query vector is multiplied by the linear output data to obtain a point set attention output, the element query vector is multiplied by the linear output data to obtain an element attention output, and the point set attention output and the element attention output are added to obtain an attention output, and finally, grouped convolution is used to linearly map the query vector to obtain mapping data, and the mapping data is added to the attention output to obtain an attention result.

[0051] The obtained query vector is dimensionally compressed to obtain a point set query vector and an element query vector, and then matrix multiplication is performed to obtain linear output data. The point set query vector is multiplied by the linear output data to obtain the point set attention output, and the element query vector is multiplied by the linear output data to obtain the element attention output. The two are then added to obtain the attention output, thereby quickly coupling the decoupled point set and element, and calculating the attention output based on the linear output. Only one calculation of attention is required, and no additional calculation amount is added when calculating the attention. The whole process not only realizes the decoupling between the map element instance and the point set, but also has a small amount of calculation. It avoids the disadvantage of needing to execute the attention module multiple times, and also avoids the problem of excessive complexity caused by long sequences. While ensuring the attention calculation effect, its efficiency is significantly improved, reducing the computational complexity and improving the computational efficiency.

[0052] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The description of the following exemplary embodiments is actually merely illustrative and is by no means intended to limit the present invention and its application or use. The techniques and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, the techniques and equipment should be considered as part of the specification. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0053] Figure 1A flowchart of a method for quickly calculating attention of map elements provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0054] S1: Obtain the query vector, key vector and value vector of the map element input into the preset transformer model;

[0055] S2: performing dimension compression on the query vector to obtain a point set query vector and an element query vector, and performing linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0056] S3: multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output;

[0057] S4: Use grouped convolution to linearly map the query vector to obtain mapping data, and add the mapping data to the attention output to obtain an attention result.

[0058] Step S1 is the process of obtaining the query vector, key vector and value vector about the map element input into the transformer model.

[0059] To facilitate the explanation of this embodiment, for example, the input of the transformer model is Q (Query, query data), K (Key, key data), and V (Value, numerical data). In order to reduce the amount of calculation and facilitate the convolution operator, Q, K, and V need to be transformed respectively.

[0060] Specifically, in a specific embodiment, the step of obtaining a query vector, a key vector, and a value vector about a map element input into a preset transformer model includes:

[0061] S11: obtaining query data, key data and numerical data about map elements input into a preset transformer model;

[0062] S12: performing matrix transformation on the query data, key data and numerical data to respectively obtain a query vector, a key vector and a numerical vector having four-dimensional matrix dimensions.

[0063] That is, first obtain Q, K, and V, and then perform matrix conversion on Q, K, and V, all of which are converted into the form of four-dimensional matrix dimensions, such as the form of [B, d, N, M], starting from 0, B represents the zeroth dimension (also called the zeroth dimension), d represents the first dimension (also called the first dimension), N and M represent the second dimension (also called the second dimension) and the third dimension (also called the third dimension), respectively. In this embodiment, B represents how many pictures (pictures about the map) can be read simultaneously in the subsequent reasoning process. In the subsequent process, the value is generally 1, that is, one picture is read each time; d is the feature dimension, for example, a map contains N elements, each of which contains M points, and each point is represented by a feature vector, and d represents the feature dimension of the feature vector; N represents the number of elements contained in the map, that is, how many elements are there in the map; M represents the number of points contained in each element, that is, how many points are there in each element. It should be noted that the conversion of query data, key data and numerical data into query vectors, key vectors and numerical vectors with four-dimensional matrix dimensions is only for the convenience of convolution calculation. There is no substantial change. Therefore, the converted query vectors, key vectors and numerical vectors are still represented by Q, K and V in the following text.

[0064] Then a specific decoupling method based on dimensional compression is performed. Step S2 is to perform dimensional compression on the query vector to obtain a point set query vector and an element query vector, and to perform linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and to perform matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data.

[0065] Figure 2 The logical diagram of a specific embodiment of the method for fast attention calculation of the entire map element is shown in the figure. In the figure, step S2 is further divided into two steps: S2A: dimensionally compressing the query vector to obtain a point set query vector and an element query vector; S2B: linearly mapping the key vector and the numerical vector to obtain a key linear output and a numerical linear output respectively; S2C: performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0066] exist Figure 2 There are three dotted boxes, one on the left and two on the right. Figure 2 The dotted box on the left is called the left picture, and the upper and lower dotted boxes on the right are called the upper right picture and the lower right picture respectively.

[0067] Step S2A is a process of compressing the dimension of the query vector to obtain a point set query vector and an element query vector. In a specific embodiment, in order to achieve fast decoupling of elements and point sets and to avoid executing multiple attention modules, after obtaining the matrix dimension of the query vector Q, the dimension of the query vector Q is directly compressed, as shown in the attached figure. Figure 2 As shown in the dashed box in the lower right figure, Q is decoupled into a point set query vector query Q1 for point sets and an element query vector queryQ2 for elements. The decoupling strategy is as follows:

[0068] In this embodiment, the step of performing dimension compression on the query vector to obtain a point set query vector includes:

[0069] S2AA1: Get the matrix dimension of the query vector;

[0070] S2AA2: taking the mean over a second dimension of the matrix dimension of the query vector to obtain a first mean;

[0071] S2AA3: Perform dimension transformation on the first mean to obtain a point set query vector.

[0072] exist Figure 2 In the more specific embodiment shown, for Q, the mean is first taken on the second dimension (the dimension starts from 0) through mean, then connected to convolution and relu activation function, and then dimension transformation is performed, and the output is Q1. The calculation process is as follows:

[0073] 1) Take the mean: Q_pts = relu(conv2d(mean(Q, dim = 2, keepdim = True)), the specific parameters of the mean, convolution and relu activation functions are not limited, as long as the above steps can be performed. In a more specific embodiment, the convolution kernel of the conv2d is 1x1xdx(n*dim), n is the number of attention heads, dim is the dimension of each attention head, and d == n*dim. The shape of Q_pts is [B, d, 1, M]; so as to facilitate convolution operations and further improve the calculation rate;

[0074] 2) Dimension transformation: via the attached Figure 2 The reshape in the dotted box in the lower right figure shows that Q1 = reshape(Q_pts,(B*n,dim,M)), where the shape of Q1 after dimensionality transformation is [B*n,dim,M].

[0075] Similarly, the step of performing dimension compression on the query vector to obtain an element query vector includes:

[0076] S2AB1: Obtain the matrix dimension of the query vector;

[0077] S2AB2: taking the mean over the third dimension of the matrix dimension of the query vector to obtain a second mean;

[0078] S2AB3: Perform dimension transformation on the second mean to obtain an element query vector.

[0079] exist Figure 2 In the more specific embodiment shown, for Q, the average is taken on the third dimension (the dimension starts from 0), and the processing process is similar to the above, and the output is Q2. The calculation process is as follows:

[0080] 1) Take the mean: Q_ins = relu(conv2d(mean(Q, dim = 3, keepdim = True))). The specific parameters of the mean, convolution and relu activation functions are not limited, as long as the above steps can be performed. In a more specific embodiment, the convolution kernel of conv2d is 1x1xdx(n*dim), n is the number of attention heads, dim is the dimension of each attention head, and d == n*dim. The shape of Q_ins is [B, d, N, 1]; so as to facilitate convolution operations and further improve the calculation rate;

[0081] 2) Dimension transformation: Q2 = reshape(Q_ins,(B*n,dim,N)), the shape of Q2 is [B*n,dim,N].

[0082] In this way, the elements and point sets can be quickly decoupled, while avoiding the execution of multiple attention modules, reducing the amount of calculation and complexity.

[0083] Step S2B is a process of linearly mapping the key vector and the numerical vector respectively to obtain a key linear output and a numerical linear output, and performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data.

[0084] Specifically, the step of linearly mapping the key vector to obtain a key linear output includes:

[0085] S2BA1: Input the key vector into a preset first linear processing unit for convolution activation linear processing to obtain a key linear vector; wherein the first linear processing unit includes a preset convolution layer and an activation function layer, the convolution kernel of the convolution layer is 1x1xdx(n*dim), and the activation function layer adopts a relu activation function;

[0086] S2BA2: deform the key linear vector to obtain a key linear output.

[0087] The step of linearly mapping the numerical vector to obtain a numerical linear output comprises:

[0088] S2BB1: Perform convolution linear processing on the numerical vector input value in a second linear processing unit preset to obtain a numerical linear vector; wherein the second linear processing unit includes a convolution layer; and the convolution kernel of the convolution layer is 1x1xdx(n*dim);

[0089] S2BB2: Transform the numerical linear vector to obtain a numerical linear output.

[0090] Then, S2C is performed immediately: a process of performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data, wherein the step of performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data includes:

[0091] S2C1: multiply the matrix of the key linear outputs by the matrix of the numerical linear outputs to obtain a linear product;

[0092] S2C2: Divide the linear product by a preset control variance function to obtain linear output data.

[0093] Specifically, the specific embodiments of the above-mentioned S2BA, S2BB and S2C processes can be abstractly represented as follows: Figure 2 The part of the process within the dashed box in the middle left figure.

[0094] In this specific embodiment, steps S2BA and S2BB can be performed simultaneously without any sequence constraints. Specifically, the specific steps of the method for rapidly decoupling attention are as follows:

[0095] Use a convolution kernel of 1x1xdx(n*dim) to linearly map K, then connect the relu activation function, and then transform it into [B*n,dim,N*M] based on reshae and permute;

[0096] At the same time, a convolution with a convolution kernel of 1x1xdx(n*dim) is used to perform linear mapping on V, and then it is transformed into [B*n,dim,N*M] based on reshape.

[0097] Then perform S2C, which uses matrix multiplication to calculate VK T Since V is [B*n, dim, N*M] and K is [B*n, dim, N*M], then VK T The shape is [B*n, dim, dim], in order to control VK T The variance of VK needs to be controlled TThe variance of the control variance function (sqrt(dim)*N*M) is used as the final linear output data; the essence of the linear output data is still the matrix product of the above V and K after linear mapping, but it is divided by the control variance function for the control variance, but its essence has not changed, so the linear output data is still VK in the following text T express.

[0098] So far, the logic of the whole process of step S2 is to decouple Q into query Q1 for the point set and query Q2 for the element, and do not decouple K and V, but directly calculate their product. At this time, the sequence length is N*M, and the subsequent control VK T The variance of , and then divided by (sqrt(dim)*N*M), where, since linear attention does not have a softmax function, dividing by sqrt(dim) is equivalent to QK in standard self-attention T / sqrt(dim), the whole process of step S2 is attached Figure 2 The dotted boxes in the left figure and the dotted boxes in the lower right figure are the processes shown in the figure. It should be noted that the dotted boxes in the left figure (decoupling Q into query Q1 for point sets and query Q2 for elements) and the dotted boxes in the lower right figure (linearly processing K and V and obtaining the product of the two and then dividing by (sqrt(dim)*N*M)) are not in order and can be performed simultaneously. As long as these two steps can be completed in time, the calculation basis for subsequent processes can be provided. Step S2 directly compresses the dimension of Q to achieve rapid decoupling of elements and point sets, and avoids executing multiple attention modules. At the same time, a linear attention method is introduced to perform linear calculations on K and V. Subsequently, the attention between elements and the attention between point sets can be quickly calculated based on this to avoid the problem of excessive complexity.

[0099] After executing step S2, the comprehensive calculation step S3 can be executed. Step S3 is a process of multiplying the point set query vector with the linear output data to obtain the point set attention output, multiplying the element query vector with the linear output data to obtain the element attention output, and adding the point set attention output and the element attention output to obtain the attention output.

[0100] This process is specifically manifested as Figure 2 The part shown in the dashed box in the upper right figure is the point set query vector Q1 and the linear output data VK T Matrix multiplication, get the point set attention output context1, element query vector Q2 and linear output data VK T Multiply the matrices to get the element attention output context2, and then add context1 and context2 to get the attention output context.

[0101] In the process of obtaining the point set attention output context1, the product of the two is first calculated, and then the final point set attention output context1 is obtained by reshaping.

[0102] The calculation process is as follows:

[0103] a.context1 = matmul(VK T ,Q1). Among them, the output shape is [B*n,dim,M];

[0104] b.context1=reshape(context1,(B,d,1,M));

[0105] In the process of obtaining the element attention output context2, the product of the two is first calculated, and then it is transformed through reshape to obtain the final element attention output context1;

[0106] The calculation process is as follows:

[0107] a.context2 = matmul(VK T ,Q2). Among them, the output shape is [B*n,N,dim];

[0108] b.context2=reshape(context2,(B,d,1,N));

[0109] Finally, context1 and context2 are added to get context, and the shape of context is [B, d, N, M];

[0110] At this point, the logic of the entire process is to directly calculate the product of K and V without decoupling them. At this time, the sequence length is N*M. T The variance of , and then divided by (sqrt(dim)*N*M), as shown in the dashed box on the left. Since linear attention does not have a softmax function, dividing by sqrt(dim) is equivalent to QK in the standard self-attention T / sqrt(dim); then use the above query Q1 and query Q2 to perform matrix multiplication with the value after dividing by this value to obtain the attention output of the point set and the element. Finally, to couple the element and the point set, the two attention outputs are added together to obtain the total attention output context.

[0111] Finally, step S4 is performed. Step S4 is a process of linearly mapping the query vector using grouped convolution to obtain mapping data, and adding the mapping data to the attention output to obtain an attention result. The specific implementation example is shown in the attached figure. Figure 2 The dashed box in the upper right figure and the attached Figure 2 The rightmost part of the arrow outside the box, as shown in the attached Figure 2 As shown in the figure, a group convolution DWC is used to map Q to obtain the mapping data, which is then added to the attention output context to obtain the final output attention result out.

[0112] Where, out = context + DWC (Q);

[0113] The DWC can be any component and convolution layer with the above functions, as long as it can meet the above functions. In this specific embodiment, the convolution kernel of the DWC is 1x1xdxdim, and the grouping parameter is n, so as to facilitate accurate calculation and reduce complexity.

[0114] In short, the entire calculation process from step S1 to step S4 is as shown in the attached figure. Figure 2 As shown, the details are as follows:

[0115] 1. The input of transformer is Q, K, V. To facilitate the use of convolution operators, the shapes of Q, K, and V are converted to [B, d, N, M], where B is the batch size, which is generally 1 during inference, and d is the feature dimension;

[0116] 2. Directly compress the dimension of Q and decouple it into query Q1 for point sets and query Q2 for elements. The decoupling strategy is as follows:

[0117] (1) For Q, take the mean value in the second dimension (the dimension starts from 0, N), then connect the convolution and relu activation function, and then perform dimension transformation, and the output is Q1. The calculation process is as follows:

[0118] 1) Q_pts = relu(conv2d(mean(Q, dim = 2, keepdim = True)), where the convolution kernel of conv2d is 1x1xdx(n*dim), n is the number of attention heads, dim is the dimension of each attention head, d == n*dim. The shape of Q_pts is [B, d, 1, M];

[0119] 2) Q1=reshape(Q_pts,(B*n,dim,M)), the shape of Q1 is [B*n,dim,M].

[0120] (2) For Q, take the mean value in the third dimension (the dimension starts from 0, M). The processing process is similar to a, and the output is Q2. The calculation process is as follows:

[0121] 1) Q_ins = relu(conv2d(mean(Q, dim = 3, keepdim = True))), where the convolution kernel of conv2d is 1x1xdx(n*dim), n is the number of attention heads, dim is the dimension of each attention head, d == n*dim. The shape of Q_ins is [B, d, N, 1];

[0122] 2) Q2=reshape(Q_ins,(B*n,dim,N)), the shape of Q2 is [B*n,dim,N].

[0123] 3. K and V are not decoupled, and their product is directly calculated after linear mapping. At this time, the sequence length is N*M, and it is the variance of the control. The product is divided by (sqrt(dim)*N*M) to obtain the final VK T ; Use query Q1 and queryQ2 with VK respectively T Perform matrix multiplication to obtain the attention output of the point set and element, and then add the two together for subsequent operations, as follows:

[0124] 1) Use a convolution kernel of 1x1xdx(n*dim) to linearly map K, then connect the relu activation function, and then transform it into [B*n,dim,N*M];

[0125] 2) Use a convolution kernel of 1x1xdx(n*dim) to linearly map V, and then transform it into [B*n,dim,N*M];

[0126] 3) Calculate VK using matrix multiplication T , whose shape is [B*n, dim, dim], and then divided by (sqrt(dim)*N*M) to get the VK used later T ;

[0127] 4) Combine Q1 and VK T Matrix multiplication outputs context1. The calculation process is as follows:

[0128] a.context1 = matmul(VK T ,Q1). Among them, the output shape is [B*n,dim,M];

[0129] b.context1=reshape(context1,(B,d,1,M));

[0130] 5) Combine Q2 with VK T Matrix multiplication outputs context2. The calculation process is as follows:

[0131] a.context2 = matmul(VK T ,Q2). Among them, the output shape is [B*n,N,dim];

[0132] b.context2=reshape(context2,(B,d,1,N));

[0133] 6) Add context1 and context2 to get context, whose shape is [B, d, N, M];

[0134] 7) Use a group convolution DWC to map Q, and then add it to the context to get the final attention result out.

[0135] out=context+DWC(Q).

[0136] Among them, the convolution kernel of DWC is 1x1xdxdim, and the grouping parameter is n.

[0137] The logic of the whole process is to decouple Q into query Q1 for point sets and query Q2 for elements, while K and V are not decoupled and their product is directly calculated. At this time, the sequence length is N*M. At the same time, the subsequent control VK T The variance of , and then divided by (sqrt(dim)*N*M), as shown in the dashed box on the left. Since linear attention does not have a softmax function, dividing by sqrt(dim) is equivalent to QK in the standard self-attention T / sqrt(dim); then use the above query Q1 and queryQ2 and the value after dividing by this value to perform matrix multiplication to obtain the attention output of the point set and the element. Finally, in order to couple the element and the point set, the two attention outputs are added to obtain the total attention output context. This not only realizes the decoupling between the map element instance and the point set, but also has a relatively small amount of calculation, avoids the disadvantage of needing to execute the attention module multiple times, and also avoids the problem of high complexity caused by long sequences. While ensuring the calculation effect, its efficiency can be significantly improved.

[0138] As described above, the method for fast attention calculation of map elements provided by the present invention first performs dimension compression on the obtained query vector to obtain a point set query vector and an element query vector, then performs matrix multiplication to obtain linear output data, and then multiplies the point set query vector by the linear output data to obtain a point set attention output, multiplies the element query vector by the linear output data to obtain an element attention output, and then adds the two to obtain an attention output, thereby quickly coupling the decoupled point set and element, and calculating the attention output based on the linear output, only requiring one calculation of the attention, and while calculating the attention, no additional calculation amount is added. The whole process not only realizes the decoupling between the map element instance and the point set, but also has a small amount of calculation, which not only avoids the disadvantage of needing to execute the attention module multiple times, but also avoids the problem of excessive complexity caused by long sequences. While ensuring the attention calculation effect, its efficiency is significantly improved, reducing the computational complexity and improving the computational efficiency.

[0139] A map element rapid attention calculation system provided in an embodiment of the present application is introduced below. The map element rapid attention calculation system described below and the map element rapid attention calculation method described above can refer to each other.

[0140] like Figure 3 As shown, the present invention also provides a map element fast attention calculation system 100, which can implement the map element fast attention calculation method as described above, and the system includes:

[0141] A dimension vector acquisition unit 110, used to acquire a query vector, a key vector and a value vector about a map element input into a preset transformer model;

[0142] A dimension compression unit 120, configured to perform dimension compression on the query vector to obtain a point set query vector and an element query vector;

[0143] A linear mapping unit 130, configured to perform linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and perform matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0144] A linear coupling unit 140, multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output;

[0145] The result output unit 150 is used to use grouped convolution to linearly map the query vector to obtain mapping data, and add the mapping data to the attention output to obtain an attention result.

[0146] The dimension vector acquisition unit 110 includes a data acquisition module 111 and a data conversion module 112, wherein:

[0147] The data acquisition module 111 is used to acquire query data, key data and numerical data about map elements input into the preset transformer model;

[0148] The data conversion module 112 is used to perform matrix conversion on the query data, key data and numerical data to obtain a query vector, a key vector and a numerical vector having four-dimensional matrix dimensions respectively.

[0149] The dimension compression unit 120 includes a point set processing module 121 and an element processing module 122, wherein:

[0150] The point set processing module 121 is used to obtain the matrix dimension of the query vector; take the mean on the second dimension of the matrix dimension of the query vector to obtain a first mean; perform dimension transformation on the first mean to obtain a point set query vector;

[0151] The element processing module 122 is used to obtain the matrix dimension of the query vector; take the mean on the third dimension of the matrix dimension of the query vector to obtain a second mean; perform dimension transformation on the second mean to obtain an element query vector;

[0152] The linear mapping unit 130 includes a key linear output module 131 , a numerical linear output module 132 , and a bus type output module 133 ;

[0153] The key linear output module 131 is used to input the key vector into a preset first linear processing unit for convolution activation linear processing to obtain a key linear vector; the first linear processing unit includes a preset convolution layer and an activation function layer; wherein the convolution kernel of the convolution layer is 1x1xdx(n*dim); the activation function layer adopts a relu activation function; and the key linear vector is deformed to obtain a key linear output.

[0154] The numerical linear output module 132 performs convolution linear processing on the numerical vector input value in a preset second linear processing unit to obtain a numerical linear vector; the second linear processing unit includes a convolution layer; the convolution kernel of the convolution layer is 1x1xdx(n*dim); the numerical linear vector is deformed to obtain a numerical linear output.

[0155] The bus-type output module 133 is used to multiply the matrix of the key linear output by the matrix of the numerical linear output to obtain a linear product; and divide the linear product by a preset control variance function to obtain linear output data.

[0156] The map element rapid attention calculation system 100 provided by the present invention generally performs calculations as the aforementioned map element rapid attention calculation method. For a more specific implementation process, reference may be made to the specific embodiments of the aforementioned map element rapid attention calculation method.

[0157] As described above, the map element fast attention calculation system 100 provided by the present invention performs dimension compression on the acquired query vector based on the dimension vector acquisition unit 110 and the dimension compression unit 120 to obtain the point set query vector and the element query vector, and then performs matrix multiplication calculation based on the linear mapping unit 130 to obtain linear output data, and then multiplies the point set query vector and the linear output data through the linear coupling unit 140 to obtain the point set attention output, and multiplies the element query vector and the linear output data to obtain the element attention output, and then adds the two to obtain the attention output, and finally obtains the final attention result through the result output unit 150. The whole process quickly couples the decoupled point set and element, and calculates the attention output based on the linear output, only needs to calculate the attention once, and while calculating the attention, no additional calculation amount is added. The whole process not only realizes the decoupling between the map element instance and the point set, but also has a small amount of calculation, which not only avoids the disadvantage of needing to execute the attention module multiple times, but also avoids the problem of excessive complexity caused by long sequences. While ensuring the attention calculation effect, its efficiency is significantly improved, reducing the computational complexity and improving the computational efficiency.

[0158] See also Figure 4, which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, including: a memory 11 for storing a computer program; a processor 12 for implementing the steps of the map element fast attention calculation method described in any of the above method embodiments when executing the computer program. In this embodiment, the device can be a vehicle-mounted computer, a PC (Personal Computer), or a terminal device such as a smart phone, a tablet computer, a handheld computer, a portable computer, etc. The device may include a memory 11, a processor 12, and a bus 13. Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may be an internal storage unit of the device in some embodiments, such as a hard disk of the device. The memory 11 may also be an external storage device of the device in other embodiments, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed on the device, such as program codes for executing fault prediction methods, but also can be used to temporarily store data that has been output or is to be output. In some embodiments, the processor 12 can be a central processing unit (CPU). In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code stored in the memory 11 or process data, such as the code for executing a fast attention calculation method for map elements. The bus 13 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Further, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the device and other electronic devices.

[0159] Optionally, the device may further include a user interface 15, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 15 may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface. Figure 4 Only the device with components 11-15 is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation of the device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0160] An embodiment of the present invention further provides a computer-readable storage medium, which may be non-volatile or volatile, and stores a computer program, which, when executed by a processor, implements:

[0161] Get the query vector, key vector and value vector of the map element input into the preset transformer model;

[0162] Performing dimension compression on the query vector to obtain a point set query vector and an element query vector, and performing linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data;

[0163] multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output;

[0164] The query vector is linearly mapped using grouped convolution to obtain mapping data, and the mapping data is added to the attention output to obtain an attention result.

[0165] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the embodiment of the method for fast attention calculation of map elements, which will not be repeated here.

[0166] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0167] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0169] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0170] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0171] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0172] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for methods, devices, electronic devices and computer-readable storage media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The methods, devices, electronic devices and media described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0173] As described above with reference to the accompanying drawings, the map element fast attention calculation method and system proposed in the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the map element fast attention calculation method and system proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A method for fast attention calculation of map elements, characterized in that: include: Get the query vector, key vector and value vector of the map element input into the preset transformer model; Performing dimension compression on the query vector to obtain a point set query vector and an element query vector, and performing linear mapping on the key vector and the numerical vector to obtain a key linear output and a numerical linear output, respectively, and performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data; multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output; The query vector is linearly mapped using grouped convolution to obtain mapping data, and the mapping data is added to the attention output to obtain an attention result.

2. The method for calculating the fast attention of map elements according to claim 1, characterized in that: The steps of obtaining the query vector, key vector and value vector of the map element input into the preset transformer model include: Get query data, key data and numerical data about map elements input into the preset transformer model; The query data, key data and numerical data are matrix-converted to obtain a query vector, a key vector and a numerical vector having four-dimensional matrix dimensions, respectively.

3. The method for calculating the fast attention of map elements as claimed in claim 2, characterized in that: The step of performing dimension compression on the query vector to obtain a point set query vector comprises: Obtaining the matrix dimension of the query vector; Taking the mean over a second dimension of the matrix dimensions of the query vector to obtain a first mean; The first mean is dimensionally transformed to obtain a point set query vector.

4. The method for calculating the rapid attention of map elements as claimed in claim 2, characterized in that: The step of performing dimension compression on the query vector to obtain an element query vector comprises: Obtaining the matrix dimension of the query vector; Taking the mean over a third dimension of the matrix dimensions of the query vector to obtain a second mean; The second mean is dimensionally transformed to obtain an element query vector.

5. The method for calculating the rapid attention of map elements according to claim 1, characterized in that: The step of linearly mapping the key vector to obtain a key linear output comprises: Input the key vector into a preset first linear processing unit for convolution activation linear processing to obtain a key linear vector; wherein the first linear processing unit includes a preset convolution layer and an activation function layer, the convolution kernel of the convolution layer is 1x1xdx(n*dim), and the activation function layer adopts a relu activation function; The key linear vector is deformed to obtain a key linear output.

6. The method for calculating the rapid attention of map elements according to claim 5, characterized in that: The step of linearly mapping the numerical vector to obtain a numerical linear output comprises: Performing convolution linear processing on the numerical vector input value in a second linear processing unit preset by the second linear processing unit to obtain a numerical linear vector; wherein the second linear processing unit includes a convolution layer, and the convolution kernel of the convolution layer is 1x1xdx(n*dim); The numerical linear vector is transformed to obtain a numerical linear output.

7. The method for calculating the rapid attention of map elements according to claim 6, characterized in that: The step of performing matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data comprises: multiplying the matrix of key linear outputs by the matrix of numerical linear outputs to obtain a linear product; The linear product is divided by a preset control variance function to obtain linear output data.

8. A map element fast attention calculation system, characterized in that: To implement the method for quickly calculating attention of map elements according to any one of claims 1 to 7, the system comprises: A dimension vector acquisition unit, used to acquire a query vector, a key vector and a value vector about a map element input into a preset transformer model; A dimension compression unit, used for performing dimension compression on the query vector to obtain a point set query vector and an element query vector; A linear mapping unit, used to perform linear mapping on the key vector and the numerical vector respectively to obtain a key linear output and a numerical linear output, and perform matrix multiplication calculation based on the key linear output and the numerical linear output to obtain linear output data; a linear coupling unit, multiplying the point set query vector by the linear output data to obtain a point set attention output, multiplying the element query vector by the linear output data to obtain an element attention output, and adding the point set attention output and the element attention output to obtain an attention output; A result output unit is used to linearly map the query vector using grouped convolution to obtain mapping data, and add the mapping data to the attention output to obtain an attention result.

9. An electronic device, characterized in that: The device includes: a processor, a memory and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the map element rapid attention calculation method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for fast attention calculation of map elements described in any one of claims 1-7 is implemented.

Images