Image processing method, electronic equipment and computer readable storage medium

Abstract

The invention provides an image processing method. The image processing method comprises the steps of obtaining a first BEV feature of first data; obtaining a second BEV feature of the second data; based on the first BEV feature and the second BEV feature, obtaining a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network; and performing fusion processing on the third BEV feature and the fourth BEV feature to obtain a fused feature, the first data and the second data being data collected by different sensors. The fusion performance can be improved. Related steps in the embodiment provided by the invention can be realized through an AI model. Functions associated with the AI may be performed by a non-volatile memory, a volatile memory, and a processor.

Description

Technical Field

[0001] The present invention relates to the field of image signal processing, and in particular to a technology for fusing bird's eye view (BEV) features of multiple modalities. Background Art

[0002] High-definition (HD) maps provide rich and accurate static environment information of driving scenes, which is crucial and challenging for the planning of autonomous driving systems. The construction of HD maps regards this task as the problem of predicting a set of vectorized static map elements in a bird's-eye view (BEV), such as crosswalks, lane dividers, road boundaries, etc.

[0003] Recently, multimodal fusion methods (e.g., Camera-LiDAR) have received increasing attention in high-definition map construction tasks, which can significantly improve the baseline. Although data from different modalities are mapped into a unified BEV space, due to the large semantic gap between different modalities, there is still a certain degree of inconsistency in the semantics of the lidar BEV features and the camera EBV features, resulting in semantic misalignment problems and a decrease in fusion performance.

[0004] Therefore, a technique that can achieve better fusion is needed. Summary of the invention

[0005] Therefore, according to an embodiment of the present invention, there is provided an image processing method, comprising: obtaining a first bird's-eye view (BEV) feature of first data; obtaining a second BEV feature of second data; based on the first BEV feature and the second BEV feature, obtaining a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network; and fusing the third BEV feature and the fourth BEV feature to obtain a fused feature, wherein the first data and the second data are data collected by different sensors.

[0006] In one example, a third BEV feature of first data and a fourth BEV feature of second data are obtained through a self-attention network, including: based on a BEV channel, flattening the first BEV feature to obtain a first feature matrix of the first data, and based on the BEV channel, flattening the second BEV feature to obtain a second feature matrix of the second data; cascading the first feature matrix and the second feature matrix to obtain a third feature matrix; and obtaining the third BEV feature and the fourth BEV feature based on the third feature matrix through the self-attention network.

[0007] In one example, the third BEV feature and the fourth BEV feature are obtained based on a third feature matrix through a self-attention network, including: applying at least one self-attention operation to the third feature matrix through the self-attention network; cascading the feature vectors output by each self-attention operation to obtain a cascaded feature vector; performing a nonlinear transformation on the cascaded feature vector to obtain a fourth feature matrix; and obtaining the third BEV feature and the fourth BEV feature based on the fourth feature matrix.

[0008] In one example, each self-attention operation includes: performing three linear transformations on the third feature matrix to obtain a query vector, a key vector, and a value vector, respectively; applying a self-attention operation to the query vector, the key vector, and the value vector to obtain a feature vector output by the self-attention operation.

[0009] In one example, the third BEV feature and the fourth BEV feature are fused to obtain the fused feature, including: respectively determining a first weight value corresponding to the third BEV feature and a second weight value corresponding to the fourth BEV feature; and performing fusion processing based on the third BEV feature and the first weight value, and the fourth BEV feature and the second weight value to obtain the fused feature.

[0010] In one example, respectively determining the first weight value and the second weight value includes: fusing the third BEV feature and the fourth BEV feature; performing pooling processing on the fused BEV features; and obtaining the first weight value and the second weight value based on the pooled features.

[0011] In one example, the first weight value and the second weight value are obtained based on the features after pooling processing, including: performing a first linear operation on the features after pooling processing by reducing the channel dimension; performing a second linear operation on the features after the first linear operation by increasing the channel dimension; and obtaining the first weight value and the second weight value based on the features after the second linear operation.

[0012] In one example, obtaining the fused feature includes: cascading a third BEV feature weighted by a first weight value and a fourth BEV feature weighted by a second weight value along a channel dimension to obtain a cascade feature; reducing the channel dimension of the cascade feature; pooling the cascade feature of the reduced dimension; and obtaining the fused feature based on the pooled cascade feature and the cascade feature of the reduced dimension.

[0013] In one example, the fused features are used for high-precision map reconstruction.

[0014] According to an embodiment of the present invention, a method for image processing is provided, including: obtaining a first bird's-eye view (BEV) feature of first data; obtaining a second BEV feature of second data; and fusing the first BEV feature and the second BEV feature based on a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature, respectively, to obtain a fused feature, wherein the first data and the second data are data collected by different sensors.

[0015] In one example, the fusion includes: respectively determining a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature; performing fusion processing based on the first BEV feature and the first weight value, and the second BEV feature and the second weight value to obtain the fused feature.

[0016] In one example, respectively determining the first weight value and the second weight value includes: fusing the first BEV feature and the second BEV feature; performing pooling processing on the fused BEV features; and obtaining the first weight value and the second weight value based on the pooled features.

[0017] In one example, obtaining the first weight value and the second weight value based on the features after pooling processing includes: performing a first linear operation on the features after pooling processing by reducing the channel dimension; performing a second linear operation on the features after the first linear operation by increasing the channel dimension; and obtaining the first weight value and the second weight value based on the features after the second linear operation.

[0018] In one example, obtaining a fused feature includes: cascading a first BEV feature weighted by a first weight value and a second BEV feature weighted by a second weight value along a channel dimension to obtain a cascade feature; reducing the channel dimension of the cascade feature; pooling the cascade feature of the reduced dimension; and obtaining a fused feature based on the pooled cascade feature and the cascade feature of the reduced dimension.

[0019] In one example, the method also includes, before fusing the first BEV feature and the second BEV feature: based on the first BEV feature and the second BEV feature, obtaining a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network, and fusing the first BEV feature and the second BEV feature includes fusing the third BEV feature and the fourth BEV feature to obtain a fused feature.

[0020] In one example, obtaining a third BEV feature of first data and a fourth BEV feature of second data through a self-attention network includes: flattening the first BEV feature based on a BEV channel to obtain a first feature matrix of the first data, and flattening the second BEV feature based on the BEV channel to obtain a second feature matrix of the second data; cascading the first feature matrix and the second feature matrix to obtain a third feature matrix; and obtaining the third BEV feature and the fourth BEV feature based on the third feature matrix through a self-attention network.

[0021] In one example, the third BEV feature and the fourth BEV feature are obtained based on a third feature matrix through a self-attention network, including: applying at least one self-attention operation to the third feature matrix through the self-attention network; cascading the feature vectors output by each self-attention operation to obtain a cascaded feature vector; performing a nonlinear transformation on the cascaded feature vector to obtain a fourth feature matrix; and obtaining the third BEV feature and the fourth BEV feature based on the fourth feature matrix.

[0022] In one example, in each self-attention operation, the third feature matrix is ​​linearly transformed three times to obtain a query vector, a key vector and a value vector respectively; the self-attention operation is applied to the query vector, the key vector and the value vector to obtain the feature vector output by the self-attention operation.

[0023] In one example, the fused features are used for high-precision map reconstruction.

[0024] According to an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the aforementioned methods.

[0025] According to an embodiment of the present invention, there is provided a computer-readable storage medium on which a computer program is stored, wherein the computer program implements any of the aforementioned methods when executed by a processor.

[0026] Therefore, according to the embodiments of the present invention, the information between different data can be fully utilized to achieve a significant improvement in fusion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be more readily understood through the following detailed description with the aid of the accompanying drawings, in which like reference numerals designate like structural elements, and in which:

[0028] Figure 1 Schematic diagram showing BEV characteristics of different modes;

[0029] Figure 2 A schematic diagram showing an existing method for fusing multimodal BEV features;

[0030] Figure 3 A schematic diagram showing an architecture for fusing multi-modal BEV features according to an embodiment of the present invention;

[0031] Figure 4 A schematic flow chart showing a method for image processing according to a first embodiment of the present invention;

[0032] Figure 5 A schematic diagram showing a dual-branch dynamic fusion module according to an embodiment of the present invention is shown;

[0033] Figure 6 A schematic flow chart showing a method for image processing according to a second embodiment of the present invention;

[0034] Figure 7 The qualitative result analysis according to the embodiment of the present invention is shown;

[0035] Figure 8 A schematic diagram showing an electronic device according to a first embodiment of the present invention;

[0036] Fig. 9 A schematic diagram showing an electronic device according to a second embodiment of the present invention;

[0037] Fig.10 A schematic diagram showing an electronic system according to an embodiment of the present invention; and

[0038] Fig.11 A schematic diagram showing another electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In recent years, multimodal bird's-eye view (BEV) feature fusion methods have received extensive attention.

[0041] Figure 1 Schematic diagram showing BEV characteristics of different modes.

[0042] like Figure 1As shown in , although in the same BEV space, due to the large modality gap, the LiDAR (light detection and ranging, LiDAR) BEV features and the camera BEV features may still be semantically inconsistent to some extent. Existing methods directly perform arithmetic operations or splicing operations on multimodal BEV features, resulting in semantic misalignment problems and affecting fusion performance.

[0043] Figure 2 A schematic diagram showing the existing fusion method of multimodal BEV features.

[0044] Figure 2 a in the figure shows the convolution fusion method: the BEV features of different modalities are first cascaded / connected in series, and then the convolution operation is used to fuse the BEV features from different modalities to obtain the aggregated features for high-precision map construction.

[0045] Figure 2 Figure b shows the additive fusion method: first, a convolutional network is used to convolve different BEV features, and then the convolved features are added to obtain aggregated BEV features for high-precision map construction.

[0046] Figure 2 Figure c shows a dynamic fusion method: first, the BEV features of different modules are cascaded and fused with learnable static weights. This method is inspired by the Squeeze-and-Excitation mechanism and uses a simple channel attention module to select important fusion features.

[0047] Among the three existing fusion strategies, Figure 2 The convolution fusion method a in can be used for high-precision map reconstruction, and other methods are applied to the field of 3D target detection.

[0048] There are mainly the following two problems in the related art.

[0049] First, although in the same BEV space, due to the large modality gap, LiDAR BEV features and camera BEV features may still be semantically inconsistent to some extent. Existing methods directly perform arithmetic operations or splicing operations on multimodal BEV features, resulting in semantic misalignment.

[0050] Second, existing multimodal BEV feature fusion methods use very simple fusion operations to fuse EBV features of different modalities, resulting in the problem of information loss.

[0051] Figure 3 A schematic diagram showing an architecture for fusing multi-modal BEV features according to an embodiment of the present invention is shown.

[0052] like Figure 3 As shown in , features are extracted from multimodal sensors and transformed into a shared bird’s-eye view (BEV) space using a view transformer. In order to fuse the BEV features of different modalities, a cross-modal interaction transform (CIT) module is used to enhance one modality from another modality through a cross-attention mechanism.

[0053] In addition, the architecture according to the present invention can also utilize a dual dynamic fusion (DDF) module to automatically select valuable information from different modalities to achieve better feature fusion.

[0054] Finally, the fused multimodal BEV features can be fed into the detector and prediction head for HD map construction.

[0055] Although in Figure 3 It is shown that both the CIT module and the DDF module are used in fusing multi-modal BEV features. However, according to an embodiment of the present invention, using the CIT module or the DDF module alone can utilize the information between different modalities, thereby achieving better feature fusion.

[0056] In addition, although Figure 3 The fusion of camera images and lidar images is shown in the figure, but those skilled in the art will appreciate that the embodiments of the present invention can be applied to the fusion of various data, not limited to camera images and lidar images.

[0057] In the description of the embodiments of the present invention, the terms "first", "second", "third", etc. are merely used to distinguish corresponding features, parts, components, elements, etc., but not to limit the importance of the corresponding features, parts, components, elements.

[0058] First embodiment

[0059] Figure 4 A schematic flow chart of a method 400 for image processing according to a first embodiment of the present invention is shown.

[0060] like Figure 4 As shown in , in S410 of method 400 , a first BEV characteristic of first data is obtained.

[0061] In S420 , a second BEV characteristic of the second data is obtained.

[0062] In S430, based on the first BEV feature and the second BEV feature, a third BEV feature of the first data and a fourth BEV feature of the second data are obtained through a self-attention network.

[0063] In S440, the third BEV feature and the fourth BEV feature are fused to obtain a fused feature, wherein the first data and the second data are data collected by different sensors.

[0064] According to an embodiment of the present invention, data collected by different sensors may be referred to as data of different modalities. The first data may be data of a camera image, and the second data may be data of a radar image, such as a LiDAR point cloud.

[0065] Hereinafter, description will be given by taking an example where the first data is data of a camera image and the second data is data of a radar image, but the embodiment of the present invention is not limited thereto.

[0066] Therefore, according to the embodiments of the present invention, the information between different data / modalities can be fully utilized to achieve a significant improvement in fusion performance.

[0067] For the convenience of description, some symbols and definitions used in the embodiments of the present invention are first introduced here.

[0068] Assume that multimodal sensor data χ is used as input and predict vectorized map elements in BEV space, where the set of map elements is classified into: road boundaries, lane dividers and pedestrian crossings. The input modality χ = {Camera, LiDAR} contains: multi-view RGB camera images in perspective, N cam , H cam , W cam denotes the number of cameras, image height and image width respectively; and LiDAR point cloud, LiDAR∈R P×5 , the number of points is P, and each point consists of its 3D coordinates, reflectivity and ring index.

[0069] In S410 and S420 , the camera BEV feature of the camera image and the radar BEV feature of the radar image are extracted respectively.

[0070] The camera BEV feature of the camera image is the first BEV feature, and the radar BEV feature of the radar image is the second BEV feature.

[0071] A baseline method for building high-precision maps based on multi-feature fusion based on the MapTR model can be established. Figure 3As shown in , initial features are extracted from two sensor inputs. For camera images, a 2D-3D transformation is used to encode pixel-level semantic features in the image perspective. First, a 2D convolution is performed from each input image Extracting perspective features And predict the depth distribution of D equidistant scattered points associated with each pixel in the input image, D can be 512 or other values, and the pixels correspond to the scattered points one by one. Assign to D equally spaced scattered points along the direction of the camera light to obtain D×H×W pseudo point cloud features Finally, the pseudo point cloud features are flattened to the BEV space through pooling to obtain the camera BEV features C represents the number of channels, which can be 512.

[0072] For radar features, taking LiDAR points as an example, we can follow the SECOND model and use voxelization and sparse LiDAR encoder. LiDAR features are projected into BEV space using the flatten operation in the BEVFusion model to obtain a unified BEV representation, namely, radar BEV features.

[0073] In the related art, all sensor features are directly converted into shared BEV features, and then they are fused together through arithmetic operations or splicing operations to obtain multimodal BEV features. However, despite being in the same BEV space, due to the large modality gap between LiDAR BEV features and camera BEV features, there is still a certain degree of misalignment in semantics, resulting in semantic misalignment problems.

[0074] The embodiment of the present invention proposes a cross-modal interaction Transform (CIT) module, which enhances one modality from another modality through a cross-attention mechanism. Next, CIT will be described in detail.

[0075] In S430, a third BEV feature of the first data and a fourth BEV feature of the second data are obtained through a self-attention network by using a CIT module based on the first BEV feature and the second BEV feature.

[0076] First, by transforming the camera BEV features and LiDAR BEV features The new camera feature matrix is ​​obtained by flattening the BEV channels and arranging the order of their matrices and radar signature matrix Here, the feature matrix can be a token matrix. Then, the token matrices of the two modalities are concatenated and a learnable position embedding is added, which is a trainable parameter of dimension 2HW×C. Thus, a new input BEV token matrix T is obtained. in ∈R 2HW ×C The learnable position embedding is a trainable parameter of dimension 2HW×C, which enables the model to distinguish the spatial information between different Token matrices during training. Then, for the input Token matrix T in Apply at least one self-attention operation.

[0077] For example, when inputting the Token matrix T in In each self-attention operation applied, the input Token matrix T in A set of query (Q) vectors, key (K) vectors, and value (V) vectors (Q, K, V) are calculated using linear projection, such as cubic linear transformation, as shown in equation (1) below. In the attention mechanism, query vectors (Query Vector), key vectors (Key Vector), and value vectors (Value Vector) are three basic vector representations, which are used to describe the input sequence, calculate similarity, and output weighted information, respectively.

[0078] Q=T in W Q , K = T in W K , V = T in W V (1)

[0079] in, is the weight matrix, and in the CIT module, D Q , D K and D V Can be equal, for example D Q =D K =D V =C.

[0080] After that, a self-attention operation is applied to Q, K, and V to obtain the feature vector output by this self-attention operation. Specifically, the self-attention layer uses the scaled dot product between Q and K to calculate the attention weight, which is then multiplied by the value V to infer the refined output Z, as shown in equation (2)

[0081]

[0082] in is a scaling factor used to prevent the softmax function from falling into a region of extremely small gradients when the magnitude of the dot product becomes larger.

[0083] In order to encapsulate multiple complex relationships from different representation subspaces at different positions, a multi-head attention mechanism can be adopted, as shown in Equation (3) below.

[0084]

[0085]

[0086] Where the subscript h represents the number of heads, and W O ∈R h·C×C Concat(Z 1 ,...,Z h ), i represents the index of each self-attention operation.

[0087] The self-attention model can be viewed as establishing the interaction between different forms of the input vector (Token matrix) in a linear projection space. Multi-head attention refers to establishing different projection information in multiple different projection spaces, and then performing different projections on the input matrix (Token matrix), obtaining many output matrices, and then splicing / cascading them together.

[0088] According to an embodiment of the present invention, the input Token matrix T in Apply at least one, i.e., i.e., i.e., self-attention operation, and concatenate the feature vectors outputted each time to obtain a concatenated feature vector.

[0089] Finally, the CIT module uses nonlinear transformation to calculate the output Token matrix feature T out , whose shape is similar to the input Token matrix feature T in The same is shown in the following equation (4), where MLP() represents a multi-layer perceptron.

[0090]

[0091] Output T out Converted to the updated camera BEV feature and updated radar BEV signature That is, the third BEV feature of the camera image and the fourth BEV feature of the radar image for future feature fusion. For example, the inverse operation of the cascade can be used to output T out Convert to and

[0092] The key idea of ​​the CIT module is to combine the global information of the camera modality and the lidar modality using the self-attention mechanism due to their complementary nature. Specifically, we weight each position of the input multimodal BEV features using the correlation matrix. Therefore, the CIT module can automatically perform both intra-modal and inter-modal information fusion and robustly capture the inherent complementary information between BEV features of different modalities.

[0093] By fusing the updated camera BEV features and updated radar BEV signature Better fusion features can be achieved.

[0094] In S440 , the updated camera BEV feature and the updated radar BEV feature are fused to obtain a fused feature.

[0095] For example, you can use Figure 2 The methods shown in a to c in FIG. 1 are used to fuse the updated camera BEV features and the updated radar BEV features.

[0096] Specifically, if Figure 2 The convolution fusion method shown in a in the figure can transform the updated camera BEV features and updated radar BEV signature Concatenate / cascade, then perform convolution to combine features from multimodal BEV feature inputs to obtain aggregate features F fused Alternatively, if Figure 2 The additive fusion method shown in b can use CNN to separately fusion the BEV features of different modalities. and Perform convolution and then add the convolution features to get F fused Alternatively, if Figure 2 The dynamic fusion method shown in c in , whose input can be the output features of the convolutional fusion method, and then they are fused with learnable static weights.

[0097] In addition, in addition to utilizing the CIT module to effectively improve the fusion effect, an effective cross-modal fusion strategy can also be applied to adaptively select valuable information from different modalities to achieve better feature fusion.

[0098] According to an embodiment of the present invention, inspired by the squeeze and excitation mechanism, in order to effectively select valuable information from different modalities, the present disclosure proposes a dual-branch dynamic fusion ( Dual dynamic Fusion, DDF) module to achieve better feature fusion and maximum performance gain. Next, the updated camera BEV feature (Third BEV Signature) and Updated Radar BEV Signature The DDF module is described as an example of input using the fourth BEV feature. However, those skilled in the art will appreciate that the input of the DDF module may also be the camera BEV feature. (First BEV Signature) and Radar BEV Signature (Second BEV feature).

[0099] Figure 5 A schematic diagram of a dual-branch dynamic fusion module according to an embodiment of the present invention is shown.

[0100] like Figure 5 As shown, the dual-branch dynamic fusion (DDF) module takes two sets of features from the camera BEV features and the LiDAR BEV features as input. Here, it is the updated camera BEV features and updated radar BEV signature In order to generate meaningful attention weights that can effectively select informative features from the two inputs, we first determine the updated camera BEV features, respectively. The corresponding first weight value and updated radar BEV feature The corresponding second weight value.

[0101] Specifically, the features of the two branches are summed and pooled before the squeeze and excitation operations to generate the attention weights, as shown in Equation (5) below.

[0102]

[0103] Among them, σ and γ represent the sigmoid function and linear layer respectively, and w represents the attention weight.

[0104] Then, based on and the first weight value, and The weighted camera BEV feature and the weighted radar BEV feature are obtained by multiplying w and 1-w by the two previous input features. For example, the updated camera BEV feature can be multiplied by w. And multiply the updated radar BEV signature by 1-w.

[0105] like Figure 5 As shown, when calculating the first weight value and the second weight value, the feature Fusion with features The fused BEV features are pooled, such as average pooling, and then based on the pooled features, a first weight value w and a second weight value 1-w are obtained.

[0106] According to an embodiment of the present invention, based on the features after pooling processing, a first linear operation is performed on the features after pooling processing to reduce the channel dimension, for example, reducing the channel dimension to C / r, where r can be set according to design requirements, for example, being set to 4. Then, a second linear operation is performed on the features after the first linear operation to increase the channel dimension, and the first weight value and the second weight value are obtained.

[0107] Afterwards, the weighted camera BEV features and the weighted radar BEV features are fused, for example, summed up to obtain the fused feature F fused Here, the fusion process can be made to essentially act as a self-attention mechanism that can adaptively select useful information from different BEV features, as shown in Equation (6) below.

[0108]

[0109] where [·,·] represents the concatenation / sequential operation along the channel dimension to obtain the concatenated features. concat is a static channel and spatial fusion function, implemented by a 3×3 convolutional layer, which is used to reduce the channel dimension of the cascaded feature, for example, to C. Given the input feature f adaptive The formula is shown in equation (7) below.

[0110]

[0111] Where W represents the linear transformation matrix (e.g. 1×1 convolution), f avg represents global average pooling, and σ represents the sigmoid function.

[0112] That is, the cascade features with reduced dimensionality are pooled, and based on the pooled cascade features and the cascade features with reduced dimensionality, a fused feature can be obtained.

[0113] Therefore, the DDF module can adaptively select valuable information from the two modalities for better feature fusion.

[0114] The output fusion feature F fused It will be able to be used for high-precision map construction tasks.

[0115] According to an embodiment of the present invention, model training is performed. The MapTR model is followed to train the loss function, which consists of three parts, including classification loss Point to point loss and edge direction loss Combining these loss terms together, the overall objective function can be expressed as follows in Equation (8).

[0116]

[0117] where λ 1 , 2 and λ 3 is a hyperparameter used to balance these terms. The parameters of the entire network can be updated by the stochastic gradient descent (SGD) algorithm and the chain rule.

[0118] The aforementioned training in the embodiment of the present invention is offline training.

[0119] Second embodiment

[0120] Figure 6 A schematic flow chart of a method 600 for image processing according to a second embodiment of the present invention is shown.

[0121] like Figure 6 As shown, in S610 of method 600 , a first BEV characteristic of first data is obtained.

[0122] In S620 , a second BEV characteristic of the second data is obtained.

[0123] In S630, the first BEV feature and the second BEV feature are fused based on a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature to obtain a fused feature, wherein the first data and the second data are data collected by different sensors.

[0124] Therefore, according to the embodiments of the present invention, the information between different modes can be fully utilized to achieve a significant improvement in performance.

[0125] S610 and S620 are respectively compared with the reference Figure 4 The described S410 and S420 are the same and will not be described again here.

[0126] In S630, in addition to the input of the DDF module being the camera BEV feature and radar BEV signature Rather than the updated camera BEV feature and updated radar BEV signature Other than that, the rest are the same as reference Figure 5 The process described is similar.

[0127] Therefore, according to the second embodiment of the present invention, when only the DDF module is applied, the fusion effect of the multi-modal BEV features can also be improved.

[0128] The multimodal feature extraction and model training according to the second embodiment of the present invention are the same as the multimodal feature extraction and model training according to the second embodiment of the present invention.

[0129] Figure 7 A qualitative result analysis according to an embodiment of the present invention is shown.

[0130] like Figure 7 , which shows the qualitative results analysis on the NuScenes dataset. Figure 7 An example scene of nuScenes is shown: (a) six camera images, (b) lidar point cloud, (c) real BEV vectorized HD map, (d) MapTR model prediction result; (e) BEV vectorized HD map prediction result using only the CIT module, and (f) BEV vectorized HD map prediction result of MBFusion.

[0131] As can be seen, the baseline model MapTR model predictions are highly erroneous. Using the CIT module we have been able to correct the significant errors in the baseline predictions, and the full MBFusion model further improves the accuracy.

[0132] Figure 8 A schematic diagram of an electronic device 800 according to a first embodiment of the present invention is shown. The electronic device may be a terminal, a server or any other device.

[0133] like Figure 8 As shown in , the electronic device 800 includes a first obtaining unit 810, a second obtaining unit 820, a transforming unit 830 and a fusing unit 840.

[0134] The first obtaining unit 810 is configured to obtain a first BEV feature of first data.

[0135] The second obtaining unit 820 is configured to obtain a first BEV feature of the second data.

[0136] The transformation unit 830 is configured to obtain a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network based on the first BEV feature and the second BEV feature.

[0137] The fusion unit 840 is configured to perform fusion processing on the third BEV feature and the fourth BEV feature to obtain a fusion feature.

[0138] The first data and the second data are data collected by different sensors.

[0139] In one example, the transformation unit 830 is configured to: flatten the first BEV feature based on the BEV channel to obtain a first feature matrix of the first data, and flatten the second BEV feature based on the BEV channel to obtain a second feature matrix of the second data; cascade the first feature matrix and the second feature matrix to obtain a third feature matrix; and obtain the third BEV feature and the fourth BEV feature based on the third feature matrix through a self-attention network.

[0140] In one example, the transformation unit 830 is configured to: apply at least one self-attention operation to the third feature matrix through a self-attention network; cascade the feature vectors output by each self-attention operation to obtain a cascaded feature vector; perform a nonlinear transformation on the cascaded feature vector to obtain a fourth feature matrix; and obtain the third BEV feature and the fourth BEV feature based on the fourth feature matrix.

[0141] In one example, each self-attention operation includes: performing three linear transformations on the third feature matrix to obtain a query vector, a key vector, and a value vector, respectively; applying a self-attention operation to the query vector, the key vector, and the value vector to obtain a feature vector output by the self-attention operation.

[0142] In one example, the fusion unit 840 is configured to: respectively determine a first weight value corresponding to the third BEV feature and a second weight value corresponding to the fourth BEV feature; perform fusion processing based on the third BEV feature and the first weight value, and the fourth BEV feature and the second weight value to obtain the fused feature.

[0143] In one example, the fusion unit 840 is configured to: fuse the third BEV feature and the fourth BEV feature; perform pooling on the fused BEV features; and obtain the first weight value and the second weight value based on the pooled features.

[0144] In one example, the fusion unit 840 is configured to: perform a first linear operation on the features after pooling by reducing the channel dimension; perform a second linear operation on the features after the first linear operation by increasing the channel dimension; and obtain the first weight value and the second weight value based on the features after the second linear operation.

[0145] In one example, the fusion unit 840 is configured to: cascade the third BEV feature weighted by the first weight value and the fourth BEV feature weighted by the second weight value along the channel dimension to obtain a cascade feature; reduce the channel dimension of the cascade feature; pool the cascade feature of the reduced dimension; and obtain the fused feature based on the pooled cascade feature and the cascade feature of the reduced dimension.

[0146] In one example, the fused features are used for high-precision map reconstruction.

[0147] Fig. 9 A schematic diagram of an electronic device 900 according to a second embodiment of the present invention is shown. The electronic device may be a terminal, a server or any other device.

[0148] like Fig. 9 As shown in , the electronic device 900 includes a first obtaining unit 910 , a second obtaining unit 920 and a fusion unit 940 .

[0149] The first obtaining unit 910 is configured to obtain a first BEV feature of first data.

[0150] The second obtaining unit 920 is configured to obtain a second BEV feature of the second data.

[0151] The fusion unit 930 is configured to fuse the first BEV feature and the second BEV feature based on a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature, respectively, to obtain a fused feature, wherein the first data and the second data are data collected by different sensors.

[0152] In one example, the fusion unit 930 is configured to: respectively determine a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature; perform fusion processing based on the first BEV feature and the first weight value, and the second BEV feature and the second weight value to obtain the fused feature.

[0153] In one example, the fusion unit 930 is configured to: fuse the first BEV feature and the second BEV feature; perform pooling on the fused BEV features; and obtain the first weight value and the second weight value based on the pooled features.

[0154] In one example, the fusion unit 930 is configured to: perform a first linear operation on the features after pooling by reducing the channel dimension; perform a second linear operation on the features after the first linear operation by increasing the channel dimension; and obtain the first weight value and the second weight value based on the features after the second linear operation.

[0155] In one example, the fusion unit 930 is configured to: cascade a first BEV feature weighted by a first weight value and a second BEV feature weighted by a second weight value along a channel dimension to obtain a cascade feature; reduce the channel dimension of the cascade feature; pool the cascade feature of the reduced dimension; and obtain a fused feature based on the pooled cascade feature and the cascade feature of the reduced dimension.

[0156] In one example, the electronic device 900 may further include a transformation unit (not shown), which is configured to obtain a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network based on the first BEV feature and the second BEV feature, and the fusion unit 930 is also configured to fuse the third BEV feature and the fourth BEV feature to obtain a fused feature.

[0157] In one example, the transformation unit is configured to: flatten the first BEV feature based on the BEV channel to obtain a first feature matrix of the first data, and flatten the second BEV feature based on the BEV channel to obtain a second feature matrix of the second data; cascade the first feature matrix and the second feature matrix to obtain a third feature matrix; and obtain the third BEV feature and the fourth BEV feature based on the third feature matrix through a self-attention network.

[0158] In one example, the transformation unit is configured to: apply at least one self-attention operation to the third feature matrix through a self-attention network; cascade the feature vectors output by each self-attention operation to obtain a cascaded feature vector; perform a nonlinear transformation on the cascaded feature vector to obtain a fourth feature matrix; and obtain the third BEV feature and the fourth BEV feature based on the fourth feature matrix.

[0159] In one example, in each self-attention operation, the third feature matrix is ​​linearly transformed three times to obtain a query vector, a key vector and a value vector respectively; the self-attention operation is applied to the query vector, the key vector and the value vector to obtain the feature vector output by the self-attention operation.

[0160] In one example, the fused features are used for high-precision map reconstruction.

[0161] Fig.10 FIG. 1 is a schematic diagram of an electronic system 1000 according to an embodiment of the present invention.

[0162] like Fig.10 As shown, the electronic system 1000 includes a memory 1010 and a processor 1020 .

[0163] The memory 1010 is configured to store instructions.

[0164] The processor 1020 is coupled to the memory 1010 and is configured to execute the instructions so that the electronic system 1000 executes any of the aforementioned methods.

[0165] Fig.11 A schematic diagram of another electronic device 1100 according to an embodiment of the present invention is shown. The electronic device may be a terminal, a server or any other device.

[0166] like Fig.11 As shown in , the electronic device 1100 includes a memory 1110, a processor 1120, and a computer program 1130 stored in the memory 1110, wherein the processor 1120 executes the computer program 1130 to implement any of the aforementioned methods.

[0167] According to an embodiment of the present invention, the present invention further includes a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the audio signal processing method described in any one of the preceding claims is implemented.

[0168] At least one of the multiple modules may be implemented by an AI model. Functions associated with the AI ​​may be performed by a non-volatile memory, a volatile memory, and a processor.

[0169] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors such as a central processing unit (CPU), an application processor (AP), etc., graphics processing units such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor such as a neural processing unit (NPU).

[0170] One or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0171] Here, providing by learning means making a predefined operating rule or AI model of desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system.

[0172] The AI ​​model can be composed of multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0173] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, comprising: obtaining a first bird's eye view (BEV) feature of the first data; obtaining a second BEV characteristic of second data; Based on the first BEV feature and the second BEV feature, obtaining a third BEV feature of the first data and a fourth BEV feature of the second data through a self-attention network; and The third BEV feature and the fourth BEV feature are fused to obtain a fused feature. The first data and the second data are data collected by different sensors.

2. The method according to claim 1, wherein: The third BEV feature of the first data and the fourth BEV feature of the second data are obtained by the self-attention network, including: Based on the BEV channel, flattening the first BEV feature to obtain a first feature matrix of the first data, and based on the BEV channel, flattening the second BEV feature to obtain a second feature matrix of the second data; Cascading the first characteristic matrix and the second characteristic matrix to obtain a third characteristic matrix; Through the self-attention network, based on the third feature matrix, the third BEV feature and the fourth BEV feature are obtained.

3. The method according to claim 2, wherein: The third BEV feature and the fourth BEV feature are obtained based on the third feature matrix through the self-attention network, including: Applying at least one self-attention operation to the third feature matrix through a self-attention network; Cascade the feature vectors output by each self-attention operation to obtain the cascaded feature vector; Performing a nonlinear transformation on the concatenated eigenvectors to obtain a fourth eigenmatrix; and Based on the fourth characteristic matrix, the third BEV characteristic and the fourth BEV characteristic are obtained.

4. The method according to claim 3, wherein: Each self-attention operation includes: Perform three linear transformations on the third feature matrix to obtain the query vector, key vector and value vector respectively; Apply a self-attention operation to the query vector, the key vector, and the value vector to obtain a feature vector output by the self-attention operation.

5. The method according to any one of claims 1 to 4, wherein: The third BEV feature and the fourth BEV feature are fused to obtain the fused feature, including: respectively determining a first weight value corresponding to the third BEV feature and a second weight value corresponding to the fourth BEV feature; Based on the third BEV feature and the first weight value, and the fourth BEV feature and the second weight value, a fusion process is performed to obtain the fusion feature.

6. The method according to claim 5, wherein: Determining the first weight value and the second weight value respectively includes: fusing the third BEV feature and the fourth BEV feature; Performing pooling on the fused BEV features; and Based on the features after pooling processing, the first weight value and the second weight value are obtained.

7. The method according to claim 6, wherein: Obtaining the first weight value and the second weight value based on the features after pooling processing includes: Perform the first linear operation on the pooled features by reducing the channel dimension; Performing a second linear operation on the features after the first linear operation by increasing the channel dimension; and Based on the features after the second linear operation, the first weight value and the second weight value are obtained.

8. The method according to claim 5, wherein: Obtaining the fusion feature includes: Cascading the third BEV feature weighted by the first weight value and the fourth BEV feature weighted by the second weight value along the channel dimension to obtain a cascade feature; Reduce the channel dimension of the cascaded features; Pooling the concatenated features to reduce dimensionality; and The fused features are obtained based on the pooled cascade features and the cascade features with reduced dimensions.

9. A method for image processing, comprising: obtaining a first bird's eye view (BEV) feature of the first data; obtaining a second BEV characteristic of second data; as well as The first BEV feature and the second BEV feature are fused based on a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature to obtain a fused feature. The first data and the second data are data collected by different sensors.

10. The method according to claim 9, wherein: The fusion includes: respectively determining a first weight value corresponding to the first BEV feature and a second weight value corresponding to the second BEV feature; Based on the first BEV feature and the first weight value, and the second BEV feature and the second weight value, a fusion process is performed to obtain the fusion feature.

11. The method according to claim 10, wherein: Determining the first weight value and the second weight value respectively includes: fusing the first BEV feature and the second BEV feature; Performing pooling on the fused BEV features; and Based on the features after pooling processing, the first weight value and the second weight value are obtained.

12. The method according to claim 11, wherein: Based on the features after pooling processing, obtaining the first weight value and the second weight value includes: Perform the first linear operation on the pooled features by reducing the channel dimension; Performing a second linear operation on the features after the first linear operation by increasing the channel dimension; and Based on the features after the second linear operation, the first weight value and the second weight value are obtained.

13. The method according to claim 12, wherein: The fusion features include: Cascading a first BEV feature weighted by a first weight value and a second BEV feature weighted by a second weight value along a channel dimension to obtain a cascade feature; Reduce the channel dimension of the cascaded features; Pooling the concatenated features to reduce dimensionality; and Based on the pooled cascade features and the dimensionality-reduced cascade features, fused features are obtained.

14. The method of claim 9, further comprising, before fusing the first BEV feature with the second BEV feature: Based on the first BEV feature and the second BEV feature, a third BEV feature of the first data and a fourth BEV feature of the second data are obtained through a self-attention network, and The fusing of the first BEV feature and the second BEV feature includes fusing the third BEV feature and the fourth BEV feature to obtain a fused feature.

15. The method according to claim 14, wherein: The third BEV feature of the first data and the fourth BEV feature of the second data obtained by the self-attention network include: Based on the BEV channel, flattening the first BEV feature to obtain a first feature matrix of the first data, and based on the BEV channel, flattening the second BEV feature to obtain a second feature matrix of the second data; Cascading the first characteristic matrix and the second characteristic matrix to obtain a third characteristic matrix; Through the self-attention network, based on the third feature matrix, the third BEV feature and the fourth BEV feature are obtained.

16. The method according to claim 15, wherein: The third BEV feature and the fourth BEV feature are obtained based on the third feature matrix through the self-attention network, including: Applying at least one self-attention operation to the third feature matrix through a self-attention network; Cascade the feature vectors output by each self-attention operation to obtain the cascaded feature vector; Performing a nonlinear transformation on the concatenated eigenvectors to obtain a fourth eigenmatrix; and Based on the fourth characteristic matrix, the third BEV characteristic and the fourth BEV characteristic are obtained.

17. The method according to claim 16, wherein: In each self-attention operation, Perform three linear transformations on the third feature matrix to obtain the query vector, key vector and value vector respectively; Apply a self-attention operation to the query vector, the key vector, and the value vector to obtain a feature vector output by the self-attention operation.

18. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method of any one of claims 1 to 17.

Images