Method executed by electronic equipment and electronic equipment

Abstract

The embodiment of the invention provides a method executed by electronic equipment and the electronic equipment, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a plurality of modal features associated with a target object; performing feature enhancement processing on the basis of the plurality of modal features to obtain a plurality of enhanced features, the feature enhancement processing including performing feature enhancement processing on any modal feature in the plurality of modal features to obtain a plurality of enhanced features; performing feature enhancement on any modal feature based on at least one modal feature except the any modal feature in the plurality of modal features so as to obtain an enhanced feature of the any modal feature; performing feature fusion processing based on the plurality of enhanced features to obtain a fused feature; and performing processing related to the target object based on the fused feature. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a method executed by an electronic device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Multimodal feature fusion methods have attracted increasing attention in tasks such as map building and object detection. However, due to the significant differences between different modalities, features extracted from different modalities exhibit semantic inconsistencies. For example, in map building, camera data and LiDAR data are mapped to a unified Bird's-Eye View (BEV) space. However, due to the large differences between modalities, LiDAR BEV features and camera BEV features still have a certain degree of semantic inconsistency, which leads to a reduction in the accuracy of the constructed map. Summary of the Invention

[0003] According to a first aspect of this disclosure, a method performed by an electronic device is provided, comprising: acquiring a plurality of modal features associated with a target object; performing feature enhancement processing based on the plurality of modal features to obtain a plurality of enhanced features, wherein the feature enhancement processing includes, for any modal feature among the plurality of modal features, performing feature enhancement on the any modal feature based on at least one modal feature among the plurality of modal features other than the any modal feature, to obtain an enhanced feature of the any modal feature; performing feature fusion processing based on the plurality of enhanced features to obtain a fused feature; and performing processing related to the target object based on the fused feature.

[0004] Optionally, the plurality of modal features include a first modal feature and a second modal feature, wherein feature enhancement processing is performed based on the plurality of modal features to obtain a plurality of enhanced features, including: obtaining a first enhanced feature for the first modal feature through a first neural network based on the first modal feature and the second modal feature; and obtaining a second enhanced feature for the second modal feature through a second neural network based on the first modal feature and the second modal feature.

[0005] Optionally, based on the first modality feature and the second modality feature, a first enhanced feature for the first modality feature is obtained through a first neural network, including: obtaining a first key feature and a first value feature for a first attention network unit in the first neural network based on the first modality feature; obtaining a second query feature for the first attention network unit based on the second modality feature; obtaining a first feature through the first attention network unit based on the second query feature, the first key feature, and the first value feature; and obtaining the first enhanced feature based on the first feature and the second query feature.

[0006] Optionally, the first key feature and the first value feature are obtained based on the first modality feature through a first feature mapping unit in the first neural network; the second query feature is obtained based on the second modality feature through a second feature mapping unit in the second neural network; wherein, obtaining the first enhanced feature based on the first feature and the second query feature includes: obtaining a second feature based on the first feature and the second query feature through a first normalization unit in the first neural network; obtaining a third feature based on the second feature through a first multilayer perceptron unit in the first neural network; and obtaining the first enhanced feature based on the second feature and the third feature through a second normalization unit in the first neural network.

[0007] Optionally, based on the first modality feature and the second modality feature, a second enhanced feature for the second modality feature is obtained through a second neural network, including: obtaining a second key feature and a second value feature for a second attention network unit in the second neural network based on the second modality feature; obtaining a first query feature for the second attention network unit based on the first modality feature; obtaining a fourth feature through the second attention network unit based on the first query feature, the second key feature, and the second value feature; and obtaining the second enhanced feature based on the fourth feature and the first query feature.

[0008] Optionally, the first query feature is obtained based on the first modality feature through the first feature mapping unit in the first neural network; the second key feature and the second value feature are obtained based on the second modality feature through the second feature mapping unit in the second neural network; wherein, obtaining the second enhanced feature based on the fourth feature and the first query feature includes obtaining a fifth feature based on the fourth feature and the first query feature through the third normalization unit in the second neural network; obtaining a sixth feature based on the fifth feature through the second multilayer perceptron unit in the second neural network; and obtaining the second enhanced feature based on the fifth feature and the sixth feature through the fourth normalization unit in the second neural network.

[0009] Optionally, the first neural network includes a plurality of first cross-modal interaction transformation modules, which are connected in series. Each first cross-modal interaction transformation module includes a first feature mapping unit, a first attention network unit, a first normalization unit, a first multilayer perceptron unit, and a second normalization unit. In the plurality of first cross-modal interaction transformation modules, the output of the preceding first cross-modal interaction transformation module and the second query feature output from the second neural network serve as the input of the following first cross-modal interaction transformation module. The input of the first first cross-modal interaction transformation module is the first modal feature and the second query feature output from the second neural network, and the output of the last first cross-modal interaction transformation module is the first enhanced feature.

[0010] Optionally, in the case where the second neural network includes a plurality of second cross-modal interaction transformation modules and the plurality of second cross-modal interaction transformation modules are connected in series, and each second cross-modal interaction transformation module includes a second feature mapping unit, a second attention network unit, a third normalization unit, a second multilayer perceptron unit, and a fourth normalization unit, in the plurality of first cross-modal interaction transformation modules, the output of the previous first cross-modal interaction transformation module and the second query feature output from the second cross-modal interaction transformation module corresponding to the previous first cross-modal interaction transformation module are used as the input of the next first cross-modal interaction transformation module.

[0011] Optionally, the second neural network includes a plurality of second cross-modal interaction transformation modules, which are connected in series. Each second cross-modal interaction transformation module includes a second feature mapping unit, a second attention network unit, a third normalization unit, a second multilayer perceptron unit, and a fourth normalization unit. In the plurality of second cross-modal interaction transformation modules, the output of the preceding second cross-modal interaction transformation module and the first query feature output from the first neural network serve as the input of the following second cross-modal interaction transformation module. The input of the first second cross-modal interaction transformation module is the second modal feature and the second query feature output from the second neural network, and the output of the last second cross-modal interaction transformation module is the second enhancement feature.

[0012] Optionally, in the case where the first neural network includes a plurality of first cross-modal interaction transformation modules and the plurality of first cross-modal interaction transformation modules are connected in series, and each first cross-modal interaction transformation module includes a first feature mapping unit, a first attention network unit, a first normalization unit, a first multilayer perceptron unit, and a second normalization unit, in the plurality of second cross-modal interaction transformation modules, the output of the previous second cross-modal interaction transformation module and the first query feature output from the first cross-modal interaction transformation module corresponding to the previous second cross-modal interaction transformation module are used as the input of the next second cross-modal interaction transformation module.

[0013] Optionally, the first attention network unit and / or the second attention network unit employ a multi-head attention mechanism.

[0014] Optionally, performing feature fusion processing based on the multiple enhanced features to obtain fused features includes: performing feature fusion using a third neural network based on the multiple enhanced features to obtain fused features.

[0015] Optionally, based on the plurality of enhanced features, feature fusion is performed using a third neural network to obtain fused features, including: performing cascade processing based on the plurality of enhanced features to obtain cascaded features; applying at least one of convolution operation and S-function to the cascaded features to obtain processed features; obtaining a plurality of enhanced features for feature fusion based on the processed features and the plurality of enhanced features; and performing cascade processing based on the plurality of enhanced features for feature fusion to obtain the fused features.

[0016] Optionally, the plurality of enhanced features includes a first enhanced feature and a second enhanced feature. Based on the plurality of enhanced features, a third neural network is used to perform feature fusion to obtain a fused feature, including: performing a cascaded process based on the first enhanced feature and the second enhanced feature to obtain a seventh feature; performing a first convolution operation and applying a first S-function on the seventh feature to obtain an eighth feature; performing a second convolution operation and applying a second S-function on the seventh feature to obtain a ninth feature; performing element-wise multiplication based on the first enhanced feature and the eighth feature to obtain a tenth feature; performing element-wise multiplication based on the second enhanced feature and the ninth feature to obtain an eleventh feature; and performing a cascaded process based on the tenth feature and the eleventh feature to obtain the fused feature.

[0017] Optionally, the plurality of modal features include: a first bird's-eye view BEV feature and a second BEV feature, wherein the first BEV feature is extracted from an image captured by a camera, and the second BEV feature is extracted from point cloud data acquired by radar.

[0018] According to a second aspect of this disclosure, an electronic device is provided, which may include: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform a method of an exemplary embodiment of this disclosure.

[0019] According to a third aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program or instructions, which, when executed by at least one processor, cause the at least one processor to perform a method of an exemplary embodiment of this disclosure.

[0020] According to a fourth aspect of this disclosure, a computer program product including a computer program is provided, wherein when the computer program is executed by a processor, a method for implementing exemplary embodiments of this disclosure is provided.

[0021] According to embodiments of this disclosure, semantically enhanced features can be obtained by performing feature enhancement on multiple modal features, thus solving the problem of semantic inconsistency between different modal features.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate exemplary embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0024] Figure 1 This is a flowchart of a method performed by an electronic device according to an exemplary embodiment of the present disclosure.

[0025] Figure 2 A schematic diagram illustrating feature enhancement of a first modal feature and a second modal feature according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 3 A schematic diagram illustrating feature fusion of a first enhancement feature and a second enhancement feature according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 4 A schematic diagram of multimodal feature fusion according to an exemplary embodiment of the present disclosure is shown.

[0028] Figure 5 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure.

[0029] Figure 6A schematic diagram of the structure of an electronic device to which embodiments of the present disclosure apply is shown. Detailed Implementation

[0030] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0031] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0032] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0033] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0034] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0035] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art to which this disclosure pertains. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0036] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0037] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0038] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0039] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0040] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0041] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0042] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0043] Optionally, in the context of speech or language, in the method performed by an electronic device according to this disclosure, a speech signal as an analog signal may be received via a speech input device (e.g., a microphone), and the speech portion may be converted into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model may be an artificial intelligence model. The artificial intelligence model may be processed by a dedicated artificial intelligence processor designed in a hardware architecture specified for processing the artificial intelligence model. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0044] Optionally, when dealing with the field of images or videos, in the method performed by an electronic device according to this disclosure, output data can be obtained by using image data as input data for an artificial intelligence model. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0045] Optionally, in the field of data intelligence processing, in the method performed by an electronic device according to this disclosure, during the reasoning or prediction phase, an artificial intelligence model can be used to perform prediction by using real-time input data. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to the artificial intelligence model. Reasoning and prediction are techniques for making logical inferences and predictions by determining information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0046] In this application, the artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operational rule or artificial intelligence model configured to perform desired features (or objectives) by training a basic artificial intelligence model with multiple training data using a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and neural network computation is performed by calculating the results of the previous layer and the multiple weight values.

[0047] Multimodal (e.g., camera data and LiDAR data) fusion methods are gaining increasing attention in map building tasks, as they can significantly improve benchmarks. Map building can be viewed as predicting map elements (e.g., pedestrian crossings, lane dividers, road boundaries, etc.) in a bird's-eye view (BEV) image. There are two main types of map element representation methods: vector representation, which represents map elements as curves or piecewise lines (keypoint coordinates and lines connecting keypoints); and mask representation, which assigns a mask to the areas covered by map elements, treating map building as an image segmentation problem. Vector-based maps are called high-precision maps, while masked maps are called semantic maps. Maps provide rich and accurate static environmental information about driving scenarios, which is crucial and challenging for the planning of autonomous driving systems. Although different modal data are mapped to a unified BEV space, due to significant differences between modalities, there is still a degree of semantic inconsistency between LiDAR BEV features and camera BEV features.

[0048] Furthermore, multimodal BEV feature fusion methods have also received widespread attention. For example, convolutional fusion methods first perform concatenation operations on BEV features from different modalities, and then use a convolutional network to fuse the BEV features from different modalities, thus obtaining fused features. Another example is additive fusion methods, which first use a convolutional network to perform convolution operations on BEV features from different modalities, and then add the convolutional features to obtain fused BEV features. Yet another example is dynamic fusion methods, which perform concatenation and convolution operations on BEV features from different modalities, and then use a simple channel attention module to select important fused features.

[0049] However, the methods described above, which directly perform arithmetic operations or concatenation on multimodal BEV features, can lead to semantic misalignment. Furthermore, using simple fusion operations to combine BEV features from different modalities can result in information loss.

[0050] Based on this, this disclosure proposes a multimodal feature fusion algorithm to at least solve the aforementioned semantic misalignment problem and / or information loss problem. The technical solutions of the embodiments of this disclosure and the technical effects produced by the technical solutions of this disclosure are described below through the description of several optional embodiments. It should be noted that the following embodiments can be referenced, borrowed, or combined with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be described again.

[0051] Figure 1 This is a flowchart of a method performed by an electronic device according to an exemplary embodiment of the present disclosure.

[0052] In step S101, multiple modal features associated with the target object are acquired. For example, the multiple modal features may include bird's-eye view BEV features extracted from multi-view images captured by a camera on the target object and BEV features extracted from point cloud data acquired by radar on the target object.

[0053] Taking vehicle mapping as an example, multi-view RGB camera images can be acquired using the vehicle's cameras, and LiDAR point cloud data can be obtained using the vehicle's LiDAR. Then, feature extraction is performed on both the multi-view RGB camera images and the LiDAR point cloud data to obtain BEV features in the image modality and the point cloud modality. The above example is merely illustrative, and the form and number of modalities are not limited to this.

[0054] In step S102, feature enhancement processing is performed based on multiple modal features to obtain multiple enhanced features. This feature enhancement processing can also be referred to as multi-directional cross-modal interactive transformation. In the feature enhancement processing, for any modal feature among the multiple modal features, feature enhancement is performed on that modal feature based on at least one modal feature other than that modal feature among the multiple modal features to obtain the enhanced feature of that modal feature.

[0055] As an example, in the case of two modal features (e.g., a first modal feature and a second modal feature), a first augmented feature for the first modal feature and a second augmented feature for the second modal feature can be obtained based on the two modal features through a first neural network and a second neural network, respectively.

[0056] When obtaining the first enhanced feature for the first modality feature, the first key feature and the first value feature for the first attention network unit in the first neural network can be obtained based on the first modality feature, the second query feature for the first attention network unit can be obtained based on the second modality feature, the first key feature and the first value feature can be obtained through the first attention network unit, and the first enhanced feature can be obtained based on the first feature and the second query feature.

[0057] For example, a first query feature, a first key feature, and a first value feature can be obtained based on the first modality feature through the first feature mapping unit in the first neural network; a second query feature, a second key feature, and a second value feature can be obtained based on the second modality feature through the second feature mapping unit in the second neural network; a first feature can be obtained based on the second query feature, the first key feature, and the first value feature through the first attention network unit in the first neural network; a second feature can be obtained based on the first feature and the second query feature through the first normalization unit in the first neural network; a third feature can be obtained based on the second feature through the first multilayer perceptron unit in the first neural network; and a first enhanced feature can be obtained based on the second feature and the third feature through the second normalization unit in the first neural network.

[0058] When obtaining the second enhanced feature for the second modality feature, the second key feature and the second value feature for the second attention network unit in the second neural network can be obtained based on the second modality feature. The first query feature for the second attention network unit can be obtained based on the first modality feature. Based on the first query feature, the second key feature and the second value feature, the fourth feature is obtained through the second attention network unit. The second enhanced feature is obtained based on the fourth feature and the first query feature.

[0059] For example, based on the first query feature, the second key feature, and the second value feature, a fourth feature can be obtained through the second attention network unit in the second neural network; based on the fourth feature and the first query feature, a fifth feature can be obtained through the third normalization unit in the second neural network; based on the fifth feature, a sixth feature can be obtained through the second multilayer perceptron unit in the second neural network; and based on the fifth feature and the sixth feature, a second enhanced feature can be obtained through the fourth normalization unit in the second neural network.

[0060] Figure 2 A schematic diagram illustrating feature enhancement of a first modal feature and a second modal feature according to an exemplary embodiment of the present disclosure is shown. The following description uses an example where the first modal feature and the second modal feature are respectively a camera BEV feature and a LiDAR BEV feature.

[0061] like Figure 2 As shown, camera BEV features can be... Input the first feature mapping unit to obtain the first query feature, the first key feature, and the first value feature. For example, the first feature mapping unit can... Flatten and arrange the tokens in order, add a learnable positional encoding feature, and obtain a new token matrix feature. Then, the first feature mapping unit performs a feature projection operation (e.g., using matrix multiplication) to... Transformed into the first query feature Q C ∈R (HW)×C First key feature K C ∈R (HW)×C and the first value feature V C ∈R (HW)×C .

[0062] LiDAR BEV characteristics can be used Input the second feature mapping unit to obtain the second query feature, the second key feature, and the second value feature. For example, the second feature mapping unit can... Flatten and arrange the tokens in order, add a learnable positional encoding feature, and obtain a new token matrix feature. Then, the second feature mapping unit can perform feature projection operations (implemented using matrix multiplication) to... Transformed into the second query feature Q L ∈R (HW)×C Second bond feature K L ∈R (HW)×C Second-valued feature V L ∈R (HW)×C .

[0063] The first feature can be obtained through a first attention network unit based on the second query feature, the first key feature, and the first value feature. For example, the first feature Z can be obtained by performing a cross-attention operation on the second query feature, the first key feature, and the first value feature. C As shown in equation (1) below.

[0064]

[0065] Based on the first query feature, the second key feature, and the second value feature, four features can be obtained through a second attention network unit. For example, the first feature Z can be obtained by performing a cross-attention operation on the first query feature, the second key feature, and the second value feature. L As shown in equation (2) below.

[0066]

[0067] According to another embodiment of this disclosure, in order to encapsulate multiple complex relationships from different representation subspaces at different locations, the first attention network unit and / or the second attention network unit may employ a multi-head attention mechanism. For example, a first feature is obtained by performing multi-head attention operations on the second query feature, the first key feature, and the first value feature. As shown in equations (3) and (4) below.

[0068]

[0069] Where h represents the number of heads in the first multi-head attention network, W1 o ∈R (h·C)×C express The projection matrix. Let represent the network parameters of the i-th head in the first multi-head attention network, which are projection matrices corresponding to the Q, K, and V (query, key, value) features, respectively.

[0070] After obtaining the first feature, the first feature and the second query feature can be input into the first normalization unit to obtain the second feature F2. The second feature is then input into the first multilayer perceptron (MLP) unit to obtain the third feature. Finally, the second and third features are input into the second normalization unit to obtain the first enhanced feature. As shown in equation (5) below.

[0071]

[0072] The first feature can be obtained by performing multi-head attention operations on the first query feature, the second key feature, and the second value feature. As shown in equations (6) and (7) below.

[0073]

[0074] Where h represents the number of heads in the second multi-head attention network, W2 O ∈R (h·C)×C express The projection matrix. represents the network parameters of the i-th head in the second multi-head attention network, which are projection matrices corresponding to the Q, K, and V (query, key, value) features, respectively.

[0075] After obtaining the fourth feature, the fourth feature and the first query feature can be input into the third normalization unit to obtain the fifth feature F5. The fifth feature is then input into the second multilayer perceptron (MLP) unit to obtain the sixth feature. Finally, the fifth feature and the sixth feature are input into the fourth normalization unit to obtain the second enhanced feature, as shown in equation (8) below.

[0076]

[0077] According to embodiments of this disclosure, at least one of the first to fourth normalization units may be omitted in the first neural network and / or the second neural network.

[0078] According to embodiments of this disclosure, the first neural network may include a plurality of first cross-modal interaction transformation modules, which are connected in series. Each first cross-modal interaction transformation module includes the first feature mapping unit, the first attention network unit, the first normalization unit, the first multilayer perceptron unit, and the second normalization unit described above.

[0079] In multiple first cross-modal interactive transformation modules, the output of the previous first cross-modal interactive transformation module and the second query feature output from the second neural network can be used as the input of the next first cross-modal interactive transformation module. The input of the first first cross-modal interactive transformation module is the first modal feature and the second query feature output from the second neural network, and the output of the last first cross-modal interactive transformation module is the first enhancement feature.

[0080] For example, in the case where the first neural network includes multiple first cross-modal interaction transformation modules and the second neural network includes a second cross-modal interaction transformation module, the input of each first cross-modal interaction transformation module may be a second query feature output from the second cross-modal interaction transformation module and the output of the previous first cross-modal interaction transformation module.

[0081] For example, in the case where the first neural network includes multiple first cross-modal interaction transformation modules and the second neural network includes the same number of second cross-modal interaction transformation modules, the input of each first cross-modal interaction transformation module may be the output of the previous first cross-modal interaction transformation module and the second query feature output from the second cross-modal interaction transformation module corresponding to the previous first cross-modal interaction transformation module.

[0082] The above examples are merely illustrative. In this disclosure, each first cross-modal interactive transformation module interacts with features from the second neural network regarding the second modal features.

[0083] The second neural network may include multiple second cross-modal interaction transformation modules, which are connected in series. Each second cross-modal interaction transformation module includes the second feature mapping unit, the second attention network unit, the third normalization unit, the second multilayer perceptron unit, and the fourth normalization unit described above.

[0084] In multiple second cross-modal interactive transformation modules, the output of the previous second cross-modal interactive transformation module and the first query feature output from the first neural network can be used as the input of the next second cross-modal interactive transformation module. The input of the first second cross-modal interactive transformation module is the second modal feature and the second query feature output from the second neural network, and the output of the last second cross-modal interactive transformation module is the second enhanced feature.

[0085] In the case where the first neural network includes multiple first cross-modal interaction transformation modules and the second neural network includes the same number of second cross-modal interaction transformation modules, in the multiple second cross-modal interaction transformation modules, the output of the previous second cross-modal interaction transformation module and the first query feature output from the first cross-modal interaction transformation module corresponding to the previous second cross-modal interaction transformation module are used as the input of the next second cross-modal interaction transformation module.

[0086] For example, refer to Figure 2 It can set up L first cross-modal interactive transformation modules and L second cross-modal interactive transformation modules, where L is a positive integer. Data interaction is performed between the first cross-modal interactive transformation modules and their corresponding second cross-modal interactive transformation modules to obtain a first enhanced feature for the first modality feature and a second enhanced feature for the second modality feature.

[0087] Figure 2 Only two neural networks are shown. In this disclosure, the number of neural networks can be determined based on the number of modalities. For example, in the case of three modalities of data, the corresponding augmented features can be obtained through three neural networks. When applying cross-attention, each modal feature can simultaneously reference two other modal features (such as concatenating query features obtained from two other modal features) or another modal feature.

[0088] This disclosure takes into account the complementarity between camera BEV features and LiDAR BEV features, and uses a cross-attention mechanism to combine the global information of camera BEV features and LiDAR BEV features, so that the enhanced camera BEV features and LiDAR BEV features are semantically aligned, thereby enabling automatic simultaneous intra-modal and inter-modal information fusion and robustly capturing the inherent complementary information between BEV features of different modalities, thus solving the problem of misalignment of different BEV features.

[0089] In step S103, feature fusion processing is performed based on multiple enhanced features to obtain fused features.

[0090] Based on multiple enhanced features, feature fusion can be performed through a third neural network to obtain fused features.

[0091] As an example, cascaded processing is performed based on multiple enhanced features to obtain cascaded features; at least one of convolution operation and S-function is applied to the cascaded features to obtain processed features; based on the processed features and multiple enhanced features, multiple enhanced features for feature fusion are obtained; cascaded processing is performed based on the multiple enhanced features for feature fusion to obtain fused features. Through the above operations, more effective / valuable information can be adaptively selected from features of different modalities to achieve better feature fusion, thereby solving the problem of information loss caused by feature fusion.

[0092] In the case of two enhanced features, a cascaded process can be performed based on the first and second enhanced features to obtain the seventh feature; a first convolution operation and a first S-function can be applied to the seventh feature to obtain the eighth feature; a second convolution operation and a second S-function can be applied to the seventh feature to obtain the ninth feature; element-wise multiplication can be performed based on the first and eighth enhanced features to obtain the tenth feature; element-wise multiplication can be performed based on the second and ninth enhanced features to obtain the eleventh feature; and cascaded processing can be performed based on the tenth and eleventh features to obtain the fused feature.

[0093] In this disclosure, the enhanced features used for feature fusion may include the tenth feature and the eleventh feature.

[0094] Figure 3 A schematic diagram illustrating feature fusion based on a first enhanced feature and a second enhanced feature according to an exemplary embodiment of the present disclosure is shown. The following description uses an example where the first enhanced feature and the second enhanced feature are respectively camera BEV features and LiDAR BEV features.

[0095] Reference Figure 3 In obtaining the first enhanced feature Second Enhancement Features After that, it can be done and Perform channel cascading to obtain the seventh feature.

[0096] Perform the first 3×3 convolution operation Conv on the seventh feature C The eighth feature is obtained by applying the first S-function (sigmoid function) σ.

[0097] Perform element-wise multiplication on the first and eighth enhanced features to obtain the tenth feature F. g.C As shown in equation (9) below.

[0098]

[0099] Perform a second 3×3 convolution operation on the seventh feature. L The ninth feature is obtained by applying the second S-function σ.

[0100] Perform element-wise multiplication on the second and ninth enhanced features to obtain the eleventh feature F. g.L As shown in equation (10) below.

[0101]

[0102] Then, the final fusion feature F fused By analyzing F g.C and F g.L Obtained by executing channel cascading.

[0103] The above examples are merely illustrative, and this disclosure is not limited thereto.

[0104] In step S104, processing related to the target object is performed based on the fusion features.

[0105] As an example, fused features can be used to perform map building or object detection for a target object.

[0106] Figure 4 A schematic diagram of multimodal feature fusion according to an exemplary embodiment of the present disclosure is shown. The following description uses the construction of a map for a target vehicle based on camera images and LiDAR point clouds as an example.

[0107] Reference Figure 4 It can acquire multi-view RGB images from multiple cameras on the target vehicle and LiDAR point cloud data from the LiDAR on the target vehicle. For example, multimodal sensor data χ = {Camera, LiDAR} contains multi-view RGB images in a perspective view. (N cam H cam W cam (representing the number of cameras, image height, and image width respectively) and the LiDAR point cloud ∈ R P×5 (P represents the number of points, and each point consists of its three-dimensional coordinates, reflectance, and ring index).

[0108] For multi-view RGB images, a 2D encoder can be used to encode pixel-level semantic features in the image perspective to obtain image features. Then, an image-to-BEV space converter transforms the image features to BEV space to obtain camera BEV features. For example, the image-to-BEV space converter can extract features from each input image. Extracting perspective features The depth distribution of D equidistant points associated with each pixel is predicted through 2D convolution operations; then the perspective features are... Assigning points to D points along the direction of the camera ray, resulting in pseudo-point cloud features of dimension D×H×W. Finally, the pseudo-point cloud features are flattened into the BEV space to obtain the camera's BEV features. In this article, H cam and W cam H and W represent the height and width of the RGB image directly obtained from the camera. H and W represent the height and width of the image in the BEV space, and they have different meanings. C represents the dimension of the BEV feature.

[0109] For LiDAR point clouds, 3D encoders (such as voxelization and sparse LiDAR encoders) can be used to encode the LiDAR point cloud features to obtain LiDAR features. These features are then projected into the BEV space using a point cloud-to-BEV space converter (such as performing a flattening operation) to obtain the LiDAR BEV features.

[0110] After obtaining the camera BEV features and LiDAR BEV features, they can be input into the bidirectional cross-modal interactive transformation module disclosed herein. The bidirectional cross-modal interactive transformation module enables knowledge exchange between the BEV features of the two modalities through a cross-attention mechanism. Each modal's BEV features can obtain enhancement information from the other modality and add it to the original BEV features, thereby obtaining enhanced camera BEV features and LiDAR BEV features. For example, the bidirectional cross-modal interactive transformation module can be as follows: Figure 2 As shown, a reference can be used. Figure 2 Feature enhancement is achieved through descriptive methods.

[0111] Next, the enhanced camera BEV features and LiDAR BEV features can be input into the multimodal feature thresholding aggregation module. This module adaptively selects valuable information from different modalities to obtain better fused features. For example, the multimodal feature thresholding aggregation module can... Figure 3 As shown, a reference can be used. Figure 3 Feature fusion is achieved through descriptive methods.

[0112] Finally, the fused multimodal BEV features are input into the decoder and prediction head to obtain the constructed map. Semantic maps and high-precision maps can be constructed based on the fused features obtained using the algorithm of this disclosure to meet the needs of different downstream applications.

[0113] For example, a model used to build high-precision maps can be trained using the following loss function. This loss function consists of three parts, including classification loss. Point-to-point loss and edge direction loss Combining these loss terms, the overall objective function can be expressed as:

[0114]

[0115] Here, λ1, λ2, and λ3 are hyperparameters used to balance these terms. The model is trained by minimizing the output of the overall objective function.

[0116] For models used to construct semantic maps, model training can be performed based on prediction heads (i.e., segmentation heads) and loss functions. The above examples are merely illustrative, and this disclosure is not limited thereto.

[0117] The methods performed by an electronic device according to exemplary embodiments of the present disclosure have been described above.

[0118] The electronic device according to embodiments of the present disclosure will now be briefly described. Figure 5 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. (Refer to...) Figure 5 The electronic device 1100 may include a memory 1101 and a processor 1102, wherein the processor 1102 is coupled to the memory 1101 and configured to perform any of the methods described above.

[0119] This disclosure also provides an electronic device including at least one processor, and optionally, at least one transceiver coupled to the at least one processor and / or at least one memory, the at least one processor being configured to perform steps of the method provided in any optional embodiment of this disclosure.

[0120] Figure 6 The diagram shows a schematic representation of an electronic device to which embodiments of this disclosure apply. For example... Figure 6 As shown, Figure 6 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, each of the processor 4001, memory 4003, and transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0121] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0122] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0123] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0124] The memory 4003 is used to store computer programs or executable instructions that execute the embodiments of this disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0125] This disclosure provides a computer-readable storage medium storing a computer program or instructions that, when executed by at least one processor, can perform or implement the steps and corresponding content of the aforementioned method embodiments.

[0126] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0127] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0128] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0129] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, comprising: Obtain multiple modal features associated with the target object; Based on the multiple modal features, feature enhancement processing is performed to obtain multiple enhanced features. The feature enhancement processing includes performing feature enhancement on any modal feature among the multiple modal features based on at least one modal feature among the multiple modal features other than the modal feature itself, so as to obtain the enhanced feature of the modal feature. Based on the multiple enhanced features, feature fusion processing is performed to obtain fused features; and Processing related to the target object is performed based on the fusion features.

2. The method according to claim 1, characterized in that, The multiple modal features include a first modal feature and a second modal feature. Among them, feature enhancement processing is performed based on the multiple modal features to obtain multiple enhanced features, including: Based on the first modal features and the second modal features, a first enhanced feature targeting the first modal features is obtained through a first neural network; Based on the first modality feature and the second modality feature, a second enhanced feature targeting the second modality feature is obtained through a second neural network.

3. The method according to claim 2, characterized in that, Based on the first modality feature and the second modality feature, a first enhanced feature targeting the first modality feature is obtained through a first neural network, including: Based on the first modality features, a first key feature and a first value feature are obtained for the first attention network unit in the first neural network; A second query feature for the first attention network unit is obtained based on the second modality feature; Based on the second query feature, the first key feature, and the first value feature, the first feature is obtained through the first attention network unit; The first enhanced feature is obtained based on the first feature and the second query feature.

4. The method according to claim 3, characterized in that, The first key feature and the first value feature are based on the first modality feature through the first feature mapping unit in the first neural network; The second query feature is obtained based on the second modality feature through the second feature mapping unit in the second neural network; The first enhanced feature is obtained based on the first feature and the second query feature, including: Based on the first feature and the second query feature, the second feature is obtained through the first normalization unit in the first neural network; Based on the second feature, a third feature is obtained through the first multilayer perceptron unit in the first neural network; Based on the second feature and the third feature, the first enhanced feature is obtained through the second normalization unit in the first neural network.

5. The method according to claim 2, characterized in that, Based on the first modality feature and the second modality feature, a second enhanced feature targeting the second modality feature is obtained through a second neural network, including: Based on the second modality features, a second key feature and a second value feature are obtained for the second attention network unit in the second neural network; A first query feature for the second attention network unit is obtained based on the first modality feature; Based on the first query feature, the second key feature, and the second value feature, a fourth feature is obtained through the second attention network unit; The second enhanced feature is obtained based on the fourth feature and the first query feature.

6. The method according to claim 5, characterized in that, The first query feature is obtained based on the first modality feature through the first feature mapping unit in the first neural network; The second key feature and the second value feature are obtained based on the second modality feature through the second feature mapping unit in the second neural network; The second enhanced feature is obtained based on the fourth feature and the first query feature, including... Based on the fourth feature and the first query feature, the fifth feature is obtained through the third normalization unit in the second neural network; Based on the fifth feature, a sixth feature is obtained through the second multilayer perceptron unit in the second neural network; Based on the fifth and sixth features, the second enhanced feature is obtained through the fourth normalization unit in the second neural network.

7. The method according to claim 4, characterized in that, The first neural network includes multiple first cross-modal interaction transformation modules, which are connected in series. Each first cross-modal interaction transformation module includes a first feature mapping unit, a first attention network unit, a first normalization unit, a first multilayer perceptron unit, and a second normalization unit. In the plurality of first cross-modal interactive transformation modules, the output of the previous first cross-modal interactive transformation module and the second query feature output from the second neural network are used as the input of the next first cross-modal interactive transformation module, and the input of the first first cross-modal interactive transformation module is the first modal feature and the second query feature output from the second neural network, and the output of the last first cross-modal interactive transformation module is the first enhanced feature.

8. The method according to claim 7, characterized in that, In the case where the second neural network includes multiple second cross-modal interaction transformation modules connected in series, and each second cross-modal interaction transformation module includes a second feature mapping unit, a second attention network unit, a third normalization unit, a second multilayer perceptron unit, and a fourth normalization unit, In the plurality of first cross-modal interactive transformation modules, the output of the previous first cross-modal interactive transformation module and the second query feature output from the second cross-modal interactive transformation module corresponding to the previous first cross-modal interactive transformation module are used as the input of the next first cross-modal interactive transformation module.

9. The method according to claim 6, characterized in that, The second neural network includes multiple second cross-modal interaction transformation modules, which are connected in series. Each second cross-modal interaction transformation module includes a second feature mapping unit, a second attention network unit, a third normalization unit, a second multilayer perceptron unit, and a fourth normalization unit. In the plurality of second cross-modal interaction transformation modules, the output of the previous second cross-modal interaction transformation module and the first query feature output from the first neural network are used as the input of the next second cross-modal interaction transformation module, and the input of the first second cross-modal interaction transformation module is the second modal feature and the second query feature output from the second neural network, and the output of the last second cross-modal interaction transformation module is the second enhancement feature.

10. The method according to claim 9, characterized in that, In the case where the first neural network includes multiple first cross-modal interaction transformation modules and the multiple first cross-modal interaction transformation modules are connected in series, and each first cross-modal interaction transformation module includes a first feature mapping unit, a first attention network unit, a first normalization unit, a first multilayer perceptron unit, and a second normalization unit, In the plurality of second cross-modal interaction transformation modules, the output of the previous second cross-modal interaction transformation module and the first query feature output from the first cross-modal interaction transformation module corresponding to the previous second cross-modal interaction transformation module are used as the input of the next second cross-modal interaction transformation module.

11. The method according to at least one of claims 3 to 10, characterized in that, The first attention network unit and / or the second attention network unit employ a multi-head attention mechanism.

12. The method according to claim 1, characterized in that, Based on the multiple enhanced features, feature fusion processing is performed to obtain fused features, including: Based on the multiple enhanced features, feature fusion is performed through a third neural network to obtain fused features.

13. The method according to claim 12, characterized in that, Based on the aforementioned multiple enhanced features, feature fusion is performed through a third neural network to obtain fused features, including: Cascaded processing is performed based on the multiple enhanced features to obtain cascaded features; At least one of convolution operation and S-function is applied to the cascaded features to obtain the processed features; Based on the processed features and the multiple enhanced features, multiple enhanced features for feature fusion are obtained; The cascaded processing is performed based on the multiple enhanced features used for feature fusion to obtain the fused features.

14. The method according to claim 12, characterized in that, The plurality of enhancement features includes a first enhancement feature and a second enhancement feature. Among them, based on the multiple enhanced features, feature fusion is performed through a third neural network to obtain fused features, including: A seventh feature is obtained by performing cascaded processing based on the first and second enhanced features; The seventh feature is subjected to a first convolution operation and the first S-function is applied to obtain the eighth feature; Perform a second convolution operation on the seventh feature and apply a second S-function to obtain the ninth feature; Based on the first enhanced feature and the eighth feature, perform element-wise multiplication to obtain the tenth feature; Based on the second and ninth enhanced features, perform element-wise multiplication to obtain the eleventh feature; The cascaded process is performed based on the tenth feature and the eleventh feature to obtain the fused feature.

15. The method according to claim 1, characterized in that, The multiple modal features include: a first bird's-eye view BEV feature and a second BEV feature, wherein the first BEV feature is extracted from an image captured by a camera, and the second BEV feature is extracted from point cloud data acquired by radar.

16. An electronic device comprising: At least one processor; as well as At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, they cause the at least one processor to perform the method as described in any one of claims 1 to 15.

17. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one processor, it causes the at least one processor to perform the method as described in any one of claims 1 to 15.

Images