A data processing method and related device

Abstract

The embodiment of the application discloses a data processing method, which is applied to a multi-modal fusion scene, and comprises the following steps: acquiring first data and second data, the modalities of the first data and the second data being different; acquiring a first feature set of the first data and a second feature set of the second data; replacing a first target feature in the first feature set with a second target feature in the second feature set to obtain a third feature set, the second target feature corresponding to the first target feature; and acquiring data features based on the third feature set and the second feature set, the data features being used to realize a computer vision task. By using the features between different modal data for replacement, the information of different modal data can be efficiently fused, so that the acquired data features have the characteristics of multi-modal data, and the expression capability of the data features is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a data processing method and related equipment. Background Art

[0002] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0003] Given that different modal data, such as text, images, and audio and video, possess varying degrees of knowledge expressivity, it is necessary to research feature representation and learning methods for information from different modalities to achieve collaborative representation of multimodal data. In the era of deep learning, multimodal feature fusion techniques have become even more important. For example, the perception systems of autonomous vehicles have been significantly improved. To achieve more robust and accurate perception results, a vehicle with assisted or autonomous driving capabilities typically requires the use of different sensors that complement each other under different operating conditions. Typical sensor modalities include cameras, radar, lidar, and high-precision maps.

[0004] At present, the strategy adopted by multimodal fusion is to combine the inputs of different modalities, input them into the same transformer structure, and obtain the final output.

[0005] However, the strategy adopted by the above multimodal fusion is simply to splice the input and is not applicable to all scenarios of multimodal fusion. Summary of the Invention

[0006] The embodiments of the present application provide a data processing method and related devices. By replacing features between different modal data, the information of different modal data can be efficiently integrated, so that the acquired data features have the characteristics of multimodal data and the expressiveness of the data features is improved.

[0007] The first aspect of the embodiment of the application provides a data processing method, which is applied to a multi-modal fusion scene, and the method comprises the following steps: obtaining first data and second data, the modalities of the first data and the second data being different; obtaining a first feature set of the first data and a second feature set of the second data; replacing a first target feature in the first feature set with a second target feature in the second feature set to obtain a third feature set, the second target feature corresponding to the first target feature; and obtaining data features based on the third feature set and the second feature set, the data features being used to implement a computer vision task. The correspondence between the second target feature and the first target feature can be determined according to a spatial relationship, a semantic relationship or the like between the first data and the second data, or can be determined according to the positions of the features in the feature sets, and the specific manner of determining the correspondence between the features in different feature sets is not limited herein.

[0008] In the embodiment of the application, the information of different modal data can be efficiently fused by replacing the features between different modal data, so that the obtained data features have the characteristics of multi-modal data, and the expression ability of the data features is improved.

[0009] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the data features based on the third feature set and the second feature set comprises the following steps: replacing a third target feature in the first feature set with a fourth target feature in the second feature set to obtain a fourth feature set, the third target feature corresponding to the fourth target feature; and obtaining the data features based on the third feature set and the fourth feature set.

[0010] In this possible implementation manner, not only the second target feature is used to replace the first target feature, but also the third target feature is used to replace the fourth target feature, so as to realize the exchange of the features between the first feature set and the second feature set. The third feature set can have the features of the modal data corresponding to the second feature set, and the fourth feature set can have the features of the modal data corresponding to the first feature set, thereby improving the expression ability of the data features generated based on the third feature set and the fourth feature set, and improving the accuracy and / or precision of the computer vision task result.

[0011] Optionally, in a possible implementation manner of the first aspect, before the step of replacing the first target feature in the first feature set with the second target feature in the second feature set, the method further comprises the following steps: obtaining a first score set of the first feature set, the first features in the first feature set corresponding to the first scores in the first score set in a one-to-one manner; obtaining a second score set of the second feature set, the second features in the second feature set corresponding to the second scores in the second score set in a one-to-one manner; and determining the second target feature based on the first score set and / or the second score set.

[0012] In this possible implementation, the second target feature or the first target feature is determined by introducing a score of the feature, which can be an index for judging the importance of the feature (for example, the greater the better), or an index for evaluating the invalidity of the feature (for example, the smaller the better), and the like. In this way, an unimportant feature in one modal data can be replaced by an important feature in another modal data, so as to improve the expression of the feature set in which the replaced feature is located for the modal data.

[0013] Optionally, in a possible implementation of the first aspect, the step of obtaining the first score set of the first feature set includes: evaluating each feature in the first feature set based on a scoring network to obtain the first score set, the scoring network being used to evaluate the importance of the feature; and the step of obtaining the second score set of the second feature set includes: evaluating each feature in the second feature set based on the scoring network to obtain the second score set.

[0014] In this possible implementation, the importance of the feature is evaluated by introducing the scoring network, so that the second target feature and the first target feature determined subsequently are more reasonable.

[0015] Optionally, in a possible implementation of the first aspect, an output value of the scoring network is subject to a sparse distribution. That is, the output value of the scoring network is more sparse, so that the score of some features is greatly different from the score of other features, and it is determined which features are useful or useless. For example, the scoring network can be trained using an L1 norm in a training process.

[0016] In this possible implementation, the score of some features is greatly different from the score of other features, and it is determined which features are useful or useless.

[0017] Optionally, in a possible implementation of the first aspect, the step of obtaining the first score set of the first feature set includes: performing mathematical operation on each first feature in the first feature set to obtain the first score set, the mathematical operation being operation based on the first feature itself, and the mathematical operation including rank operation or modulus operation; and the step of obtaining the second score set of the second feature set includes: performing mathematical operation on each second feature in the second feature set to obtain the second score set.

[0018] In this possible implementation, the mathematical operation of the feature itself is used to reduce other structures for introducing the judgment score, and to simplify the overall network structure.

[0019] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the first feature set of the first data and the second feature set of the second data based on the neural network comprises: obtaining the first feature set and the second feature set based on a neural network, the neural network comprising an attention network, a multi-layer perception, a pooling layer, or a convolution layer.

[0020] In this possible implementation manner, the first feature set and the second feature set are obtained based on the neural network, and can be applied to scenarios such as an attention network, a multi-layer perception, a pooling layer, or a convolution layer.

[0021] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the first feature set and the second feature set based on the neural network comprises: splitting the first data to obtain a plurality of first sub-data; splitting the second data to obtain a plurality of second sub-data; and inputting the plurality of first sub-data and the plurality of second sub-data into the neural network to obtain the first feature set and the second feature set.

[0022] In this possible implementation manner, the input of the neural network is obtained by splitting the modal data, so that the number of features in the subsequent obtained feature set is related to the number of splitting, and the subsequent calculation process is controlled.

[0023] Optionally, in a possible implementation manner of the first aspect, the step of replacing the first target feature in the first feature set with the second target feature in the second feature set comprises: replacing the first target feature with the second target feature based on residual position encoding, the residual position encoding being used to determine the positions of the features in the first feature set and the second feature set.

[0024] In this possible implementation manner, the position of the replaced feature is determined based on the residual position encoding, so that the position of the replaced feature in the original feature set is not changed.

[0025] Optionally, in a possible implementation manner of the first aspect, the neural network further comprises a first network layer, and a structure of the first network layer is related to the neural network.

[0026] In this possible implementation manner, the first feature set and the second feature set can be the output of the first network layer, that is, no matter where the first feature set and the second feature set are located in the neural network, the expression ability of the subsequent data features can be improved through the replacement between the features of different modal data.

[0027] Optionally, in a possible implementation manner of the first aspect, the step further comprises: inputting the data features into a second network layer to obtain a result of the computer vision task, the second network layer being related to the computer vision task.

[0028] In the possible implementation, the data feature can obtain a result of the computer vision task through the second network layer. Since the data feature is obtained by replacing the features between different modalities, the result is more accurate.

[0029] Optionally, in a possible implementation of the first aspect, the computer vision task is a classification task, and the second network layer is a fully connected layer; or the computer vision task is a segmentation task or a detection task, and the second network layer is a convolutional neural network layer or an up-sampling layer.

[0030] In the possible implementation, the method can be applied to computer vision tasks in different scenarios, and can accurately complete detection tasks, segmentation tasks, classification tasks, and the like.

[0031] The second aspect of the embodiments of the present application provides a data processing device. The data processing device is applied to a multi-modal fusion scene. The data processing device comprises: an obtaining unit, configured to obtain first data and second data. The modalities of the first data and the second data are different. The obtaining unit is further configured to obtain a first feature set of the first data and a second feature set of the second data. A replacing unit is configured to replace a first target feature in the first feature set with a second target feature in the second feature set to obtain a third feature set. The second target feature corresponds to the first target feature. The obtaining unit is configured to obtain a data feature based on the third feature set and the second feature set. The data feature is used to implement a computer vision task.

[0032] Optionally, in a possible implementation of the second aspect, the obtaining unit is specifically configured to replace a third target feature in the first feature set with a fourth target feature in the second feature set to obtain a fourth feature set. The third target feature corresponds to the fourth target feature. The obtaining unit is specifically configured to obtain the data feature based on the third feature set and the fourth feature set.

[0033] Optionally, in a possible implementation of the second aspect, the obtaining unit is further configured to obtain a first score set of the first feature set. A first feature in the first feature set corresponds to a first score in the first score set in a one-to-one manner. The obtaining unit is further configured to obtain a second score set of the second feature set. A second feature in the second feature set corresponds to a second score in the second score set in a one-to-one manner. The data processing device further comprises a determining unit configured to determine the second target feature based on the first score set and / or the second score set.

[0034] Optionally, in a possible implementation manner of the second aspect, the obtaining unit is specifically configured to evaluate each feature in the first feature set based on a scoring network to obtain the first score set, the scoring network being configured to evaluate the importance of the feature; and the obtaining unit is specifically configured to evaluate each feature in the second feature set based on the scoring network to obtain the second score set.

[0035] Optionally, in a possible implementation manner of the second aspect, an output value of the scoring network is subject to a sparse distribution.

[0036] Optionally, in a possible implementation manner of the second aspect, the obtaining unit is specifically configured to perform mathematical operation on each first feature in the first feature set to obtain the first score set, the mathematical operation being an operation based on the first feature itself, and the mathematical operation includes rank operation or modulus operation; and the obtaining unit is specifically configured to perform mathematical operation on each second feature in the second feature set to obtain the second score set.

[0037] Optionally, in a possible implementation manner of the second aspect, the obtaining unit is specifically configured to obtain the first feature set and the second feature set based on a neural network, the neural network including an attention network, a multi-layer perception, a pooling layer, or a convolution layer.

[0038] Optionally, in a possible implementation manner of the second aspect, the obtaining unit is specifically configured to split the first data to obtain a plurality of first sub-data; the obtaining unit is specifically configured to split the second data to obtain a plurality of second sub-data; and the obtaining unit is specifically configured to input the plurality of first sub-data and the plurality of second sub-data into the neural network to obtain the first feature set and the second feature set.

[0039] Optionally, in a possible implementation manner of the second aspect, the replacing unit is specifically configured to replace the first target feature with the second target feature based on residual position encoding, the residual position encoding being configured to determine the position of each feature in the first feature set and the second feature set.

[0040] Optionally, in a possible implementation manner of the second aspect, the neural network further includes a first network layer, a structure of the first network layer being related to the neural network.

[0041] Optionally, in a possible implementation manner of the second aspect, the obtaining unit is further configured to input the data feature into a second network layer to obtain the result of the computer vision task, the second network layer being related to the computer vision task.

[0042] Optionally, in a possible implementation manner of the second aspect, the computer vision task is a classification task, and the second network layer is a full connection layer; or the computer vision task is a segmentation task or a detection task, and the second network layer is a convolutional neural network layer or an up-sampling layer.

[0043] The third aspect of the embodiments of the present application provides a data processing device, comprising: a processor coupled with a memory, the memory being used to store programs or instructions, when the programs or instructions are executed by the processor, the data processing device implements the method in the first aspect or any possible implementation manner of the first aspect.

[0044] The fourth aspect of the embodiments of the present application provides a computer readable medium, which stores computer programs or instructions, when the computer programs or instructions are run on a computer, the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0045] The fifth aspect of the embodiments of the present application provides a computer program product, when the computer program product is executed on a computer, the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0046] The technical effects brought by the second, third, fourth, fifth aspects or any possible implementation manner thereof can be referred to the technical effects brought by the first aspect or different possible implementation manners of the first aspect, which will not be repeated here.

[0047] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages: by using the features between different modal data for replacement, the information of different modal data can be efficiently fused, so that the obtained data features have the characteristics of multi-modal data, and the expression ability of the data features is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The structural schematic diagram of the system architecture provided by the embodiments of the present application is shown in the figure;

[0049] Figure 2 The chip hardware structural schematic diagram provided by the embodiments of the present application is shown in the figure;

[0050] Figure 3A The structural schematic diagram of the data processing system provided by the embodiments of the present application is shown in the figure;

[0051] Figure 3B The structural schematic diagram of the data processing system provided by the embodiments of the present application is shown in the figure;

[0052] Figure 4 The flowchart of the data processing method provided by the embodiments of the present application is shown in the figure;

[0053] Figure 5A An example diagram of first data provided for embodiments of the present application;

[0054] Figure 5B An example diagram of second data provided for embodiments of the present application;

[0055] Figure 6A Another example diagram of first data provided for embodiments of the present application;

[0056] Figure 6B Another example diagram of second data provided for embodiments of the present application;

[0057] Figure 7A An example diagram of first data provided for embodiments of the present application;

[0058] Figure 7B An example diagram of second data provided for embodiments of the present application;

[0059] Figure 8A Another example diagram of first data provided for embodiments of the present application;

[0060] Figure 8B Another example diagram of second data provided for embodiments of the present application;

[0061] Figure 9 Several example diagrams of neural networks provided for embodiments of the present application;

[0062] Figure 10A An example diagram of locations of feature sets in a neural network provided for embodiments of the present application;

[0063] Figure 10B Another example diagram of locations of feature sets in a neural network provided for embodiments of the present application;

[0064] Figure 11 An example flowchart of a data processing method provided for embodiments of the present application;

[0065] Figure 12 Another example flowchart of a data processing method provided for embodiments of the present application;

[0066] Figure 13 A structural schematic diagram of a data processing device provided for embodiments of the present application;

[0067] Figure 14 Another structural schematic diagram of a data processing device provided for embodiments of the present application. DETAILED DESCRIPTION

[0068] The embodiment of the present application provides a data processing method and related equipment. By using features between different modal data for replacement, information of different modal data can be efficiently fused, so that the obtained data features have the characteristics of multi-modal data, and the expression ability of the data features is improved.

[0069] Multimodal Fusion is responsible for combining information of multiple modalities to make target prediction (classification or regression), and belongs to one of the earliest research directions of MMML. It is also the most widely used direction at present, and it also has other common aliases, such as Multi-source Information Fusion and Multi-sensor Fusion. After entering the deep learning era, the technology of multi-modal feature fusion is more important. For example, the perception system of an autonomous vehicle has been greatly improved. In order to obtain more robust and accurate perception results, a vehicle with assisted driving or autonomous driving function usually needs to be equipped with different sensors to complement each other in different working conditions. Typical sensor modalities include cameras, radars, lidar, high-precision maps, etc. At present, the strategy of multi-modal fusion is to splice the inputs of different modalities and input them into the same transformer structure to obtain the final output.

[0070] However, the strategy of the above multi-modal fusion is only to simply splice the input, and is not suitable for all scenarios of multi-modal fusion.

[0071] To solve the above technical problems, the embodiment of the present application provides a data processing method. On the one hand, by applying the transformer structure to the lane line detection task, the long-range connection between the lane lines can be effectively modeled. On the other hand, by adding the position information of the detection box of the object in the image in the process of lane line detection, the perception ability of the scene can be improved. The misjudgment in the scene where the lane line is blocked by the vehicle is reduced. The data processing method and related equipment of the embodiment of the present application will be described in detail below in combination with the drawings.

[0072] In order to facilitate understanding, the related terms and concepts mainly involved in the embodiment of the present application will be introduced first.

[0073] 1. Neural network

[0074] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with X s and intercept 1 as input. The output of the operation unit can be:

[0075]

[0076] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Relu function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0077] The operation of each layer in a neural network can be described mathematically by the expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. Bending. Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. Specifically, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.

[0078] 2. Convolutional Neural Networks

[0079] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor composed of convolutional layers and sub-sampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as making the same trainable filter convolve with an input image or a convolutional feature map. A convolutional layer refers to a layer of neurons in a CNN that performs convolutional processing on an input signal. In a convolutional layer of a CNN, a neuron can only be connected to some of the neurons in the adjacent layer. A convolutional layer usually contains several feature maps, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature map share weights, and the shared weights are the convolutional kernels. The shared weights can be understood as the way of extracting image information regardless of the location. The implicit principle is that the statistical information of a certain part of the image is the same as that of other parts. That is, the image information learned in a certain part can also be used in another part. Therefore, the same learned image information can be used for all locations on the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more the number of convolutional kernels, the more image information the convolution operation reflects.

[0080] The convolutional kernels can be initialized in the form of a matrix of random size, and the convolutional kernels can obtain reasonable weights through learning in the training process of the CNN. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the CNN and to reduce the risk of overfitting.

[0081] 3、transformer

[0082] The transformer structure is a feature extraction network (similar to a CNN) containing an encoder and a decoder.

[0083] Encoder: learn features such as pixel features in a global receptive field through self-attention.

[0084] Decoder: learn features of required modules such as output frame features through self-attention and cross-attention.

[0085] The attention (which can also be referred to as attention mechanism) is described as follows:

[0086] Attention mechanism can quickly extract important features of sparse data. Attention mechanism occurs between encoder and decoder, or between input sentence and generated sentence. However, self-attention mechanism in self-attention model occurs within input sequence or output sequence, which can extract the relationship between distant words in the same sentence, such as syntactic features (phrase structure). Self-attention mechanism provides an effective modeling method for capturing global context information through QKV. Assuming that the input is Q (query) and the context is stored in the form of key-value pair (K, V), the attention mechanism is actually a mapping function from query to a series of key-value pairs. The essence of attention function can be described as a mapping from query to a series of (key, value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), attention can complete the addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, that is, the weight, and then the weighted sum is obtained. Finally, the feature value is obtained.

[0087] The calculation of attention mainly includes three steps. The first step is to calculate the similarity between query and each key to obtain the weight. Common similarity functions include dot product, concatenation, perception, etc. Then the second step is to normalize these weights using a softmax function (on the one hand, it can be normalized to obtain a probability distribution whose sum of all weight coefficients is 1. On the other hand, the weight of important elements can be highlighted using the characteristics of the softmax function). Finally, the weighted sum of the weight and the corresponding key value value is obtained to obtain the final feature value. The specific calculation formula can be as follows:

[0088]

[0089] Where d is the dimension of QK matrix.

[0090] In addition, attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. However, the input of QKV in cross-attention is inconsistent. Attention uses the similarity (such as inner product) between features as weight to integrate the queried features as the updated value of the current feature. Self-attention is the attention extracted based on the attention of the feature map itself.

[0091] For convolution, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often needing multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its attention is global, and it can obtain the global spatial information of the feature map through simple query and assignment. The special point of self-attention in the query, key, and value (QKV) model is that the input corresponding to QKV is consistent. The QKV model will be described later.

[0092] 4. Feedforward neural network

[0093] A feedforward neural network (FNN) is the earliest invented simple artificial neural network. In a feedforward neural network, each neuron belongs to a different layer. The neurons of each layer can receive signals from the neurons of the previous layer and produce signal outputs to the next layer. The 0th layer is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. There is no feedback in the entire network, and the signal propagates unidirectionally from the input layer to the output layer.

[0094] 5. Multilayer perceptron (MLP)

[0095] A multilayer perceptron, also known as a multilayer perceptron, is a feedforward artificial neural network model that maps inputs to a single output.

[0096] 6. Loss function

[0097] During the training of a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the difference between the predicted value of the current network and the truly intended target value can be compared, and the weight vector of each layer of the neural network can be updated according to the difference between the two, (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer in the deep neural network are pre-configured), for example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the truly intended target value. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value" in advance, which is the loss function or objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0098] 7. Up-sampling

[0099] In the field of deep learning applied in computer vision, after the input image is extracted by the convolutional neural network (CNN), the output size will often be smaller, and sometimes we need to restore the image to its original size for further calculation (for example: image semantic segmentation). This operation of expanding the image size to realize the mapping of the image from small resolution to large resolution is called upsample.

[0100] Among them, there are three common methods of upsample: bilinear interpolation, transposed convolution and unpooling.

[0101] 8, modal, multi-modal, multi-modal data, multi-modal fusion

[0102] Generally speaking, modality refers to the way things happen or exist, and multi-modal refers to the combination of various forms of two or more modalities.

[0103] Each source or form of information can be called a modality. At present, the research field mainly deals with image, text, voice and other modalities.

[0104] The above-mentioned modalities can also be understood as "senses", that is, the channels through which organisms receive information with the help of sensory organs and experience. For example: humans have visual, auditory, tactile, taste and olfactory modalities. Multi-modal can be understood as the fusion of multiple senses. For example, humans can communicate with intelligent devices through sound, body language, information carriers (such as text, pictures, audio, video, etc.), environment, etc. After the intelligent device fuses multi-modal information, it makes an intention judgment for humans and feeds back to humans through text, sound, light strip and other ways.

[0105] Multi-modal data refers to data of multiple modalities, and modalities can include text, images, audio and video, etc. It can be understood that in some scenarios, images of different structures can also be referred to as different modalities, for example, RGB images and depth images are different modalities of data. Different structures of text can also be referred to as different modalities, for example, Chinese and English are different modalities of data. Different formats of audio can also be referred to as different modalities, for example, waveform sound file (MAV) and audio video interleaved format (AVI) are different modalities of data, etc.

[0106] Multi-modal fusion in deep learning refers to a technology that machines obtain information from multiple fields such as text, image, voice, and video, realize information conversion and fusion, and thus improve the performance of a model. The reason for fusing modalities is that different modalities have different ways of expression and different angles of viewing things, so there are some cross (so there is information redundancy) and complementary (so better than single feature) phenomena, and there may be multiple different information interactions between modalities. If the multi-modal information can be reasonably processed, rich feature information can be obtained.

[0107] The system architecture provided by the embodiments of the present application will be introduced below.

[0108] Referring to FIG. 1, Figure 1 The embodiments of the present application provide a system architecture 100. As shown in the system architecture 100, a data collection device 160 is configured to collect training data, and the training data in the embodiments of the present application includes data of multiple different modalities. The modalities can be text, image, audio and video. For example, the training data can include RGB images + depth images, or RGB images and point cloud data, etc. The training data is stored in a database 130, and a training device 120 trains a target model / rule 101 based on the training data maintained in the database 130. The training device 120 will be described in more detail below, which obtains the target model / rule 101 based on the training data. The target model / rule 101 can be used to implement a computer vision task to which the data processing method provided by the embodiments of the present application is applied. The computer vision task can include a classification task, a segmentation task, a detection task, or an image generation task, etc. The target model / rule 101 in the embodiments of the present application can specifically include a self-attention network, a multi-layer perception, a pooling layer, etc. It should be noted that in actual applications, the training data maintained in the database 130 can not all be collected by the data collection device 160, but can also be received from other devices. In addition, it should be noted that the training device 120 can not completely train the target model / rule 101 based on the training data maintained in the database 130, but can also obtain training data from the cloud or other places to train the model. The above description should not be regarded as a limitation of the embodiments of the present application.

[0109] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as an execution device 110 as shown in FIG. 1. Figure 1 The execution device 110 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) device / virtual reality (VR) device, a vehicle-mounted terminal, etc. Of course, the execution device 110 can also be a server or a cloud, etc. In FIG. 1, Figure 1In the middle, the execution device 110 is configured with an I / O interface 112 for data interaction with external devices, and the user can input data to the I / O interface 112 through the client device 140. The input data in the embodiment of the present application can include: to-be-detected images. In addition, the input data can be input by the user, or uploaded by the user through a shooting device, and of course it can also come from a database, which is not limited here.

[0110] The preprocessing module 113 is used for preprocessing according to the input data received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 can be used to split the input data to obtain a sub-data set. For example: the input image is an image, and the preprocessing module 113 is used to split the image to obtain a plurality of image blocks.

[0111] During the preprocessing of the input data by the execution device 110, or during the processing of the calculation module 111 of the execution device 110, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0112] Finally, the I / O interface 112 returns the processing result, such as the result corresponding to the above-mentioned target task, to the client device 140, thereby providing the user.

[0113] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different targets or different tasks. The corresponding target model / rules 101 can be used to achieve the above-mentioned target or complete the above-mentioned task, thereby providing the user with the required result.

[0114] In the attached Figure 1In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0115] It is worth noting that the Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0116] like Figure 1 As shown, the target model / rule 101 is obtained through training by the training device 120. The target model / rule 101 in the embodiment of the present application can specifically be a target neural network.

[0117] The following describes a chip hardware structure provided by an embodiment of the present application.

[0118] Figure 2 The embodiment of the present invention provides a chip hardware structure, which includes a neural network processor 20. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101.

[0119] The neural network processor 20 can be a neural-network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), among other processors suitable for large-scale XOR operation processing. Using an NPU as an example, the neural network processor 20 is mounted as a co-processor to a host central processing unit (CPU) and is assigned tasks by the host CPU. The core of the NPU is the operation circuit 203, and the controller 204 controls the operation circuit 203 to fetch data from the memory (weight memory or input memory) and perform operations.

[0120] In some implementations, the operation circuit 203 includes multiple processing engines (PEs) inside. In some implementations, the operation circuit 203 is a two-dimensional systolic array. The operation circuit 203 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 203 is a general-purpose matrix processor.

[0121] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 203 fetches the corresponding data of matrix B from the weight memory 202 and caches it on each PE of the operation circuit. The operation circuit performs matrix operations on the data of matrix A fetched from the input memory 201 and matrix B to obtain partial results or final results of the matrix, which are saved in the accumulator 208.

[0122] The vector calculation unit 207 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and the like. For example, the vector calculation unit 207 can be used for network calculation of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, and the like.

[0123] In some implementations, the vector calculation unit 207 can store the processed output vector to the unified buffer 206. For example, the vector calculation unit 207 can apply a non-linear function to the output of the operation circuit 203, such as a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 207 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the operation circuit 203, such as for use in a subsequent layer in a neural network.

[0124] The unified memory 206 is used to store input data and output data.

[0125] The weight data is directly carried to the input memory 201 and / or the unified memory 206 by the direct memory access controller (DMAC) from the external memory, the weight data in the external memory is stored in the weight memory 202, and the data in the unified memory 206 is stored in the external memory.

[0126] The bus interface unit (BIU) 210 is used to realize the interaction between the main CPU, the DMAC and the instruction fetch memory 209 through a bus.

[0127] The instruction fetch memory 209 connected with the controller 204 is used to store the instructions used by the controller 204.

[0128] The controller 204 is used to call the instructions cached in the instruction fetch memory 209 to realize the control of the working process of the operation accelerator.

[0129] Generally, the unified memory 206, the input memory 201, the weight memory 202 and the instruction fetch memory 209 are on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memories.

[0130] Next, several application scenarios of the present application are introduced.

[0131] Figure 3A A structural schematic diagram of a data processing system provided by an embodiment of the present application is shown in FIG. 1. The data processing system includes a terminal device (1001) and a server (1002). Figure 3AIn the description, only a mobile phone is used as an example of a terminal device) and a data processing device. It is understandable that, in addition to a mobile phone, the terminal device may also be a tablet computer (pad), a portable game console, a personal digital assistant (PDA), a laptop computer, an ultra mobile personal computer (UMPC), a handheld computer, a netbook, a vehicle-mounted media player, a wearable electronic device, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a vehicle, a vehicle-mounted terminal, an aircraft terminal, an intelligent robot, and other terminal devices. The terminal device is the initiator of data processing. As the initiator of a data processing request, the request is usually initiated by a user through the terminal device.

[0132] The aforementioned data processing devices can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing devices receive data processing requests from terminal devices via interactive interfaces and then perform data processing methods such as machine learning, deep learning, search, reasoning, and decision-making through the memory used to store data and the processors used for data processing. The memory in a data processing device is a general term that includes local storage and databases that store historical data. The database can be located on the data processing device or on other network servers.

[0133] exist Figure 3A In the data processing system shown, a terminal device can receive user instructions. For example, the terminal device can obtain multiple data input / selected by the user (for example, images, text, audio, etc. collected by the terminal device through the terminal device), and then initiate a request to the data processing device, causing the data processing device to execute a data processing application (for example, computer vision tasks such as classification, segmentation, detection, and image generation) on the multiple data obtained by the terminal device, thereby obtaining corresponding processing results for the multiple data. For example, the terminal device can obtain multiple images input by the user, and then initiate an image detection request to the data processing device, causing the data processing device to detect the images, thereby obtaining image detection results, and displaying the image detection results for the user to view and use.

[0134] exist Figure 3A In the embodiment of the present application, the data processing device can execute the data processing method.

[0135] Figure 3B Another structural diagram of the data processing system provided in the embodiment of the present application is shown in FIG. Figure 3B In the terminal device ( Figure 3BThe terminal device can directly obtain multiple data (for example, the data is an image, text, audio, etc.), and directly process the data by the hardware of the terminal device itself. The specific process is similar to the above description, and will not be repeated here. Figure 3A

[0136] Optionally, in the data processing system shown in Figure 3B , the terminal device can receive the user's instruction. For example, the terminal device can obtain multiple images selected by the user in the terminal device, and then execute a data processing application (for example, a computer vision task such as classification, segmentation, detection, image generation, etc.) on the images by the terminal device itself, so as to obtain the corresponding processing result of the images, and display the processing result for the user to watch and use.

[0137] Optionally, in the data processing system shown in Figure 3B , the terminal device can collect images in real time or periodically, and then execute a data processing application (for example, a computer vision task such as classification, segmentation, detection, image generation, etc.) on the images by the terminal device itself, so as to obtain the corresponding processing result of the images, and realize a function (classification function, segmentation function, detection function, image generation function, etc.) according to the processing result.

[0138] In the Figure 3B , the terminal device itself can execute the data processing method of the present application.

[0139] The terminal device in the above Figure 3A and Figure 3B may be the client device 140 or the execution device 110 in Figure 1 , the data processing device in Figure 3A may be the execution device 110 in Figure 1 , wherein the data storage system 150 can store the data to be processed of the execution device 110, and the data storage system 150 can be integrated on the execution device 110, or can be set on the cloud or other network servers.

[0140] Figure 3A The processor in Figure 3B can train / learn / deep learn data through a neural network model or other models (for example, attention model, MLP, etc.), and execute a data processing application on multiple data by using the model finally trained or learned by the data, so as to obtain the corresponding processing result.

[0141] ​The data processing method provided by the embodiments of the present application is described below. The method can be executed by a data processing device or a component (for example, a processor, a chip, or a chip system) of the data processing device. The data processing device can be a cloud device (as shown in the foregoing Figure 3A ), or a terminal device (for example, a mobile phone as shown in the foregoing Figure 3B ). Of course, the method can also be executed by a system composed of a cloud device and a terminal device (as shown in the foregoing Figure 3A ). Alternatively, the method can be processed by a CPU in the data processing device, or processed by the CPU and a GPU together, or processed without using the GPU, but using other processors suitable for neural network calculation, which is not limited in the present application.

[0142] The terminal device described above can be a mobile phone, a tablet computer (pad), a portable game console, a personal digital assistant (PDA), a notebook computer, an ultra mobile personal computer (UMPC), a handheld computer, a netbook, a car-mounted media player device, a wearable electronic device, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, and other digital products.

[0143] The application scenarios to which the method provided by the embodiments of the present application are mainly applied are multi-modal fusion scenarios, and the method can be applied to computer vision tasks such as classification scenarios, segmentation scenarios, detection scenarios, and image generation scenarios, or can be applied to semantic segmentation, indoor scene perception, outdoor driving, and the like. In addition, the data involved in the embodiments of the present application can be text, images, audio and video, and for the convenience of description, only the case that the data is an image is exemplarily described herein.

[0144] Please refer to Figure 4 , a flowchart of the data processing method provided by the embodiments of the present application is shown, and the method can include steps 401 to 404. The steps 401 to 404 are described in detail below.

[0145] In step 401, first data and second data are acquired.

[0146] In the embodiments of the present application, there are various ways for the data processing device to acquire the first data and the second data, which can be through a collection / shooting manner, or through a receiving manner of other devices, or through a selecting manner from a database, and the like, which is not limited herein.

[0147] Optionally, if applied to an autonomous driving scenario, the data processing device may be a vehicle, and the first data and the second data may be data collected by the vehicle in real time or data collected periodically, which is not specifically limited here.

[0148] In the embodiments of the present application, the first data and the second data are described as image-related data. In actual applications, the first data and the second data may also be text, audio, video, etc., which are not limited here.

[0149] In the embodiment of the present application, the relationship between the first data and the second data can be divided into multiple situations, which are described below respectively:

[0150] The first type is that the first data and the second data are isomorphic multimodal data.

[0151] Among them, homogeneous multimodal data means that the presentation method of the modality to which the first data belongs is the same as the presentation method of the modality to which the second data belongs. For example, the first data is an RGB image, and the second image is a depth image, and the presentation methods of the first data and the second data are both images.

[0152] Optionally, the first data and the second data are image data, or the first data and the second data are text data, or the first data and the second data are audio data, etc., which is not specifically limited here.

[0153] Example 1, the first data is as follows Figure 5A The RGB image shown, the second image is as Figure 5B This example 1 can be applied to cloud service scenarios (such as semantic segmentation scenarios), and the data processing device can be a smart camera, smart robot, etc.

[0154] The second type is that the first data and the second data are heterogeneous multimodal data.

[0155] Here, heterogeneous multimodal data means that the multimodality to which the first data belongs is different from the multimodality to which the second data belongs.

[0156] Optionally, the first data is image data and the second data is point cloud data, or the first data is text data and the second data is audio data, etc., which is not specifically limited here.

[0157] Example 2, the first data is as follows Figure 6A The RGB image shown, the second image is as Figure 6B Example 2 can be applied to autonomous driving scenarios (e.g., intelligent perception scenarios), and the data processing device can be a smart car, etc.

[0158] Step 402: Obtain a first feature set of the first data and a second feature set of the second data.

[0159] After the data processing device obtains the first data and the second data, it may obtain a first feature set of the first data and a second feature set of the second data.

[0160] Optionally, the first data is split to obtain a plurality of first sub-data, the second data is split to obtain a plurality of second sub-data, a first feature set is obtained based on the plurality of first sub-data, and a second feature set is obtained based on the plurality of second sub-data.

[0161] Optionally, the number of splits of the first data is related to the number of features in the first feature set. For example, the number of splits of the first data is equal to the number of features in the first feature set. Similarly, the number of splits of the second data is related to the number of features in the second feature set. For example, the number of splits of the second data is equal to the number of features in the second feature set.

[0162] In an embodiment of the present application, the rules for splitting data (including first data / second data) can be set according to actual needs. For example, the splitting rules are to divide the whole / part of the data evenly, unevenly, etc., and the specific details are not limited here.

[0163] For example, continuing the example 1 above, Figure 5A The splitting of the first data and the plurality of first sub-data can be as follows Figure 7A As shown. Figure 5B The splitting of the second data and the plurality of second sub-data can be as follows Figure 7B shown.

[0164] For example, continuing the example 2 above, Figure 6A The splitting of the first data and the plurality of first sub-data can be as follows Figure 8A As shown. Figure 6B The splitting of the second data and the plurality of second sub-data can be as follows Figure 8B As shown. Figure 8B As shown, in the case where the second data is point cloud data, the second data can be sampled to obtain sampling points, and then the sampling points are used as the second sub-data.

[0165] Optionally, after obtaining the plurality of first sub-data and the plurality of second sub-data, a first feature set can be obtained based on the plurality of first sub-data, and a second feature set can be obtained based on the plurality of second sub-data. Specifically, the first feature set of the plurality of first sub-data can be obtained based on a neural network, and the second feature set of the plurality of second sub-data can be obtained based on the neural network. Wherein, the plurality of first sub-data and the number of features in the first feature set can be one-to-one correspondence, and the plurality of second sub-data and the number of features in the second feature set can be one-to-one correspondence. Of course, in actual application, one feature can also be set to correspond to a plurality of sub-data, and a plurality of features can also be set to correspond to one sub-data, which is not limited here.

[0166] The neural network in the above can include an attention network, a multi-layer perceptron (MLP), a pooling layer, etc., which is not limited here.

[0167] For example, three structure examples of the neural network can be as shown in Figure 9 It can be understood that the neural network can also only include an attention network, a multi-layer perceptron, a pooling layer, or a convolutional layer, etc. In other words, the positions of the first feature set and the second feature set in the neural network are not limited. Taking an example of the neural network including an attention network and a first network layer, the first feature set and the second feature set can be the output of the attention network (as shown in Figure 10A ), can be the output of the first network layer (as shown in Figure 10B ), etc. (for example, the first feature set and the second feature set can be the features input into the attention network), which is not limited here. In addition, in order to facilitate subsequent example description, the attention network can include L sub-modules, or it can be understood that the attention network is an L-layer structure network, wherein the structure of each layer is the same.

[0168] In step 403, the second target feature in the second feature set is used to replace the first target feature in the first feature set.

[0169] After the data processing device obtains the first feature set and the second feature set, the second target feature in the second feature set can be used to replace the first target feature in the first feature set to obtain a third feature set. Wherein, the second target feature corresponds to the first target feature. The correspondence between the second target feature and the first target feature can be determined according to the spatial relationship, semantic relationship, etc. between the first data and the second data, or can be determined according to the position of the feature in the feature set, etc. How to determine the correspondence between different feature sets is not limited here.

[0170] Optionally, after the data processing device obtains the first feature set and the second feature set, the data processing device can first obtain a first score set of the first feature set and a second score set of the second feature set, and determine the first target feature and the second target feature using the first score set and / or the second score set, and then replace the first target feature in the first feature set with the second target feature to obtain a third feature set.

[0171] The following first describes obtaining a score set of a feature set, where the score set includes multiple scores, and the scores can be used to evaluate the importance of the features (the greater the better), or can be used to evaluate the invalidity of the features (the smaller the better), and the like. In addition, the number of scores in the score set can correspond to the number of features in the feature set. Of course, the features can also be scored in different dimensions, and in this case, one feature can correspond to multiple scores. Here, the number of scores corresponding to a feature is not limited to one or multiple. For the convenience of subsequent description, the embodiments of the present application only exemplarily describe one feature corresponding to one score.

[0172] There are many ways to obtain a score set of a feature set in the embodiments of the present application, which are described as follows:

[0173] 1. Obtain the score set corresponding to the feature set based on a scoring network.

[0174] In this way, a scoring network can be introduced, which can be used to evaluate the importance of the features.

[0175] Optionally, each feature in the first feature set is evaluated based on the scoring network to obtain the first score set. Each feature in the second feature set is evaluated based on the scoring network to obtain the second score set. Specifically, each feature in the first feature set is input into the scoring network to obtain the first score set. Each feature in the second feature set is input into the scoring network to obtain the second score set.

[0176] In addition, in order to ensure that the output value of the scoring network obeys a sparse distribution, that is, the output value of the scoring network is more sparse, so that the scores of some features are greatly different from the scores of other features, and then determine which features are useful or useless. The scoring network can be trained using L1 norm during the training process.

[0177] 2. Obtain the score set corresponding to the feature set based on mathematical operations of the features in the feature set.

[0178] The above mathematical operations can be understood as mathematical operations on the features themselves, and can include rank operation (for example, the feature is in the form of a matrix), modulus operation (for example, the feature is in the form of a vector), and the like, which are not limited here.

[0179] Optionally, in the case that the forms of the features in the first feature set and the second feature set are matrices, the score set can be obtained by performing rank operation on the feature matrices. Specifically, the rank operation is performed on each feature matrix in the first feature set to obtain the first score set. The rank operation is performed on each feature matrix in the second feature set to obtain the second score set.

[0180] Optionally, in the case that the forms of the features in the first feature set and the second feature set are vectors, the score set can be obtained by performing modulus operation on the feature vectors. Specifically, the modulus operation is performed on each feature vector in the first feature set to obtain the first score set. The modulus operation is performed on each feature vector in the second feature set to obtain the second score set.

[0181] It can be understood that the above two ways of obtaining the score set corresponding to the feature set are only examples, and in actual application, other ways of obtaining the score set can also be used, which are not limited here.

[0182] After the data processing device obtains the score set corresponding to the feature set, the first target feature and the second target feature can be determined based on the first score set and / or the second score set, and then the second target feature is used to replace the first target feature.

[0183] The corresponding relationship between the second target feature and the first target feature can be determined according to the first preset rule, and the specific implementation is not limited here. In other words, the determination of the second target feature is equivalent to the determination of the first target feature, or the determination of the first target feature is equivalent to the determination of the second target feature. Therefore, the first target feature and the second target feature can be determined based on the first score set and / or the second score set. As described above, the corresponding relationship between the second target feature and the first target feature can be determined according to the spatial relationship, semantic relationship, etc. between the first data and the second data, or can be determined according to the position of the feature in the feature set, and the specific implementation of how to determine the corresponding relationship in different feature sets is not limited here. In other words, the first preset rule can be related to the spatial relationship, semantic relationship, etc. between the multi-modal data.

[0184] Optionally, the data processing device can determine the first target feature in the first feature set based on the first score set and the second preset rule. After determining the first target feature, the second target feature corresponding to the first target feature can be determined according to the first preset rule.

[0185] Optionally, the data processing device can determine the second target feature in the second feature set based on the second score set and the second preset rule. After determining the second target feature, the first target feature corresponding to the second target feature can be determined according to the first preset rule.

[0186] The first preset rule and the second preset rule mentioned above are described as follows.

[0187] The first preset rule is specifically used for determining the correspondence between the first feature in the first feature set and the second feature in the second feature set. The correspondence can be one-to-one, one-to-many or many-to-one, which is not limited here. The first preset rule can be set according to actual needs. For example, the first preset rule includes that the feature at the first position in the first feature set corresponds to the feature at the second position in the second feature set. For another example, the first preset rule includes that the feature at the first position in the first feature set corresponds to the feature at the first position in the second feature set, and so on. In actual application, the first preset rule can also be other cases, which are not limited here. The positions of the features in the first feature set and the positions of the features in the second feature set can be determined according to residual position coding or the like.

[0188] For example, the first feature set includes A1 feature, A2 feature, A3 feature, A4 feature, A5 feature and A6 feature in turn. The second feature set includes B1 feature, B2 feature, B3 feature, B4 feature, B5 feature and B6 feature in turn. The first preset rule described above can be that A1 corresponds to B1, A2 corresponds to B2, A3 corresponds to B3, A4 corresponds to B4, A5 corresponds to B5 and A6 corresponds to B6 respectively. It can also be that A1 corresponds to B2, A2 corresponds to B3, A3 corresponds to B4, A4 corresponds to B5, A5 corresponds to B6 and A6 corresponds to B1. It can also be that A1 corresponds to B5, A2 corresponds to B3, A3 corresponds to B1, A4 corresponds to B2, A5 corresponds to B6 and A6 corresponds to B4, and so on. The first target feature is a feature in the first feature set involved in the above one correspondence, and the second target feature is a feature in the second feature set corresponding to the first target feature.

[0189] The second preset rule is specifically used for determining the first target feature and / or the second target feature. The second preset rule can be set according to actual needs. The second preset rule can be specifically related to the score size, the preset score and the like. For example, the second preset rule includes that the feature with the minimum score in the first score set is determined as the first target feature. For another example, the second preset rule includes that the feature with the maximum score in the second score set is determined as the second target feature. For another example, the second preset rule includes that the feature with the maximum score in the first score set is determined as the first target feature. For another example, the second preset rule includes that the feature with the minimum score in the second score set is determined as the second target feature. For another example, the second preset rule includes that the feature with the score equal to the preset score in the second score set is determined as the second target feature. For another example, the second preset rule includes that the feature with the score equal to the preset score in the first score set is determined as the first target feature, and so on. In actual application, the second preset rule can also be other cases, which are not limited here.

[0190] For example, assuming that the neural network comprises an attention network, the attention network can comprise L sub-modules, or be understood as an L-layer network with the same structure for each layer. The scoring network is denoted as s l The first score of the first feature (denoted as ) in the first data set in the lth layer can be denoted as The process of replacing the first target feature in the first feature set with the second target feature in the second feature set can be represented by the following formula:

[0191]

[0192] wherein, is the feature (e.g., the first target feature) to be replaced in the first feature set, and represents element-wise multiplication, is an indicator function, and the output of the indicator function is 1 if the subscript of the indicator function satisfies the condition, and the output of the indicator function is 0 if the subscript of the indicator function does not satisfy the condition. 1 can be understood as replacement, and 0 can be understood as non-replacement. θ can be understood as the preset score in the second preset rule described above, and the specific value can be set according to actual needs, for example: θ = 0.01. represents the score of the feature to be replaced in the first feature set (e.g., the score of the first target feature), represents the projection of the first target feature in the first data set to the second target feature in the second data set. A and B represent the correspondence between the features in the first feature set and the features in the second feature set (e.g., the first target feature corresponds to the second target feature ). The above formula can be understood as that the features (e.g., the first target feature) in the first feature set with a score less than θ are replaced by the corresponding features (e.g., the second target feature) in the second feature set.

[0193] Optionally, in the case of isomorphic multi-modal data where the first data and the second data have the same presentation mode, represents an identity mapping. In the case of heterogeneous multi-modal data where the first data and the second data belong to, taking the first data as an RGB image, the second data as a point cloud, and the target task as a detection task as an example. The correspondence between the image patches and the point clouds can be found by utilizing the spatial relationship between the point cloud and the image (e.g., the spatial relationship between the first preset rule and the multi-modal data described above). Specifically, it can include: assuming that there are N img image patches and N point 3D sampling points as inputs of the neural network. N point -Nimg The mapping of the Nth point Projected to the corresponding nth img The process on an image slice can be expressed as follows:

[0194]

[0195]

[0196] Among them, K and R t is the camera internal and external parameters, Represents the 3D coordinates of a point, Represents the 2D pixels of the image, W and P are the width of the original image and the width of the image slice respectively.

[0197] In addition, when there are more than two multimodal data, for example, the third data has been obtained, etc. In order to prevent the confusion of the corresponding relationship between the feature sets corresponding to different data, a distribution scheme can be fixed in advance: a B (A)∈{0,1} N ; In this case, the expression of feature replacement between multimodal data can be as follows:

[0198]

[0199] Where M is the number of different modes in the multimodal data. The rest of the explanation can refer to the description in the previous formula and will not be repeated here.

[0200] Step 404: Acquire data features based on the third feature set and the second feature set.

[0201] After the data processing device obtains the third feature set, it can obtain data features based on the third feature set and the second feature set. The data features are used to implement computer vision tasks, including classification tasks, segmentation tasks, detection tasks, or image generation tasks, etc.

[0202] In an embodiment of the present application, the position of the data feature in the neural network depends on the position of the first feature set and the second feature set. For example, the position of the data feature in the neural network can be a fusion of the positions of the first feature set and the second feature set. For another example, the position of the data feature in the neural network can also be the position of the first feature set and the second feature set pushed back one or more network layers. In the embodiment of the present application, there is no limitation on the position of the data feature in the neural network.

[0203] Exemplarily, the neural network comprises an attention network, a first network layer, and a second network layer. If the first feature set and the second feature set are outputs of the attention network, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the first network layer to obtain the data feature. If the first feature set and the second feature set are outputs of the first network layer, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the second network layer to obtain the data feature.

[0204] Exemplarily, the neural network comprises a multi-layer perceptron, a first network layer, and a second network layer. If the first feature set and the second feature set are outputs of the multi-layer perceptron, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the first network layer to obtain the data feature. The data feature can be further input into the second network layer to obtain the result of the target task. If the first feature set and the second feature set are outputs of the first network layer, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the second network layer to obtain the data feature.

[0205] Exemplarily, the neural network comprises a pooling layer, a first network layer, and a second network layer. If the first feature set and the second feature set are outputs of the pooling layer, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the first network layer to obtain the data feature. If the first feature set and the second feature set are outputs of the first network layer, obtaining the data feature based on the third feature set and the second feature set can comprise inputting the third feature set and the second feature set into the second network layer to obtain the data feature.

[0206] The second network layer described above is related to the target task and can be set according to actual needs, which is not limited here. For example, when the target task is a classification task, the second network layer can be a fully connected layer. For another example, when the target task is a segmentation task or a detection task, the second network layer can be a convolutional neural network layer or an up-sampling layer.

[0207] In addition, it should be noted that the foregoing step 403 only describes replacing the first target feature in the first feature set with the second target feature in the second feature set to obtain the third feature set. In actual applications, the fourth target feature in the second feature set can also be replaced with the third target feature in the first feature set to obtain a fourth feature set. In other words, in the data processing method provided in the embodiments of the present application, the features in the two feature sets can be replaced singly (for example, the process of obtaining the third feature set), or the features in the two feature sets can be replaced mutually (for example, the processes of obtaining the third feature set and the fourth feature set), and the specific implementation is not limited here. The process of replacing the fourth target feature in the second feature set with the third target feature in the first feature set to obtain the fourth feature set can refer to the description of the foregoing step 403, and details are not described here. The third target feature corresponds to the fourth target feature, and the specific correspondence can refer to the correspondence between the first target feature and the second target feature described above, and details are not described here.

[0208] Secondly, when the neural network includes a multi-layer structure, the feature replacement described above can be performed on at least one layer. For example, the feature replacement can be performed only on a certain layer. For another example, the feature replacement described above can be performed on multiple layers respectively. For another example, the feature replacement described above can be performed on each layer, and the specific implementation is not limited here.

[0209] Optionally, if the data processing method provided in the embodiments of the present application includes the mutual replacement of the features in the two feature sets, step 404 can include replacing the fourth target feature in the second feature set with the third target feature in the first feature set to obtain a fourth feature set. The data features are obtained based on the third feature set and the fourth feature set. The position of the third target feature in the first feature set corresponds to the position of the fourth target feature in the second feature set.

[0210] In order to more intuitively show the process of the data processing method provided in the embodiments of the present application, the following exemplary description is continued based on the foregoing examples.

[0211] For example, the foregoing example 1 is continued, and the feature set is in the positions shown in the foregoing Figure 10A , the first data is as shown in Figure 5A , the second data is as shown in Figure 5B , the first feature set and the second feature set are replaced, and the feature to be replaced is determined by the scoring network. Figure 11 For example, the foregoing example 1 is continued, and the feature set is in the positions shown in the foregoing Figure 10A , the first data is as shown in Figure 5A , the second data is as shown in Figure 5B , the first feature set and the second feature set are replaced, and the feature to be replaced is determined by the scoring network.

[0212] For example, the foregoing example 1 is continued, and the feature set is in the positions shown in the foregoing Figure 10A , the first data is as shown in Figure 5A , the second data is as shown in Figure 5B , the first feature set and the second feature set are replaced, and the feature to be replaced is determined by the scoring network. Figure 11, the neural network includes an attention network, a first network layer, and a second network layer. The first data is an RGB image, the second data is a depth image, and the first data and the second data are isomorphic multimodal data. The first data and the second image are split into 6 image slices respectively, and input into the attention network and the first network layer to obtain the first feature set (A1, A2, A3, A4, A5, A6) and the second feature set (B1, B2, B3, B4, B5, B6). The first feature set and the second feature set are scored using a scoring network trained with L1, and the features to be replaced in each feature set are determined based on the scoring values. The correspondence between the first feature set and the second feature set includes: A1-B1, A2-B2, A3-B3, A4-B4, A5-B5, A6-B6. Based on the scoring network, it is determined that the first target features in the first feature set include A2 and A6. Based on the scoring network, it is determined that the second target features in the second feature set include B2 and B6. Then use the second target feature in the second feature set to replace the first target feature in the first feature set to obtain the third feature set (A1, B2, A3, A4, A5, B6). Similarly, based on the scoring network, determine that the third target feature in the first feature set includes A3 and A4. Based on the scoring network, determine that the fourth target feature in the second feature set includes B3 and B4. Then use the third target feature in the first feature set to replace the fourth target feature in the second feature set to obtain the fourth feature set (B1, B2, A3, A4, B5, B6). After the replacement is performed to obtain the third feature set and the fourth feature set, you can enter the next layer of neural network, etc., and then input the second network layer, and fuse the output to obtain the semantic segmentation result (for example, output the result of semantic classification of each pixel). In order to ensure the accuracy of the position during feature replacement, residual position coding can be used for alignment. In addition, the above-mentioned correspondence has been described above and will not be repeated here.

[0213] For example, continuing the above example 2, the feature set is Figure 10A The position shown, the first data as Figure 6A As shown, the second data is as follows Figure 6B As shown, the first feature set and the second feature set are exchanged, and the features to be replaced are determined by the scoring network as an example. Figure 12 Provide a description.

[0214] See also Figure 12 , the neural network includes a multi-layer perceptron, a first network layer, and a second network layer. The first data is an RGB image, the second data is point cloud data, and the first data and the second data are heterogeneous multimodal data. The first data is split into 5 image slices, the second data is sampled to obtain sampling points, and the 5 image slices and the sampling points ( Figure 12In the embodiment of the present application, the sampling points are divided into 6 parts, and the input multi-layer perception and the first network layer obtain the first feature set (A1, A2, A3, A4, A5) and the second feature set (B1, B2, B3, B4, B5, B6). The scoring network trained by L1 is used to score the first feature set and the second feature set, and the features to be replaced in each feature set are determined according to the score. The corresponding relationship between the first feature set and the second feature set includes: A1-B2, A2-B6, A4-B5, A5-B3. The first target feature in the first feature set includes A1 based on the scoring network. The second target feature in the second feature set includes B2 based on the scoring network. The third feature set (B2, A2, A3, A4, A5) is obtained by replacing the first target feature in the first feature set with the second target feature in the second feature set. Similarly, the third target feature in the first feature set includes A5, A4, A2 based on the scoring network. The fourth target feature in the second feature set includes B3, B5, B6 based on the scoring network. The fourth feature set (B1, B2, A5, B4, A4, A2) is obtained by replacing the fourth target feature in the second feature set with the third target feature in the first feature set. After the third feature set and the fourth feature set are obtained by replacement, the image with the detection frame and the point cloud with the detection frame are input into the next layer of neural network and the output, respectively. In order to accurately position the features during replacement, residual position coding can be used for alignment.

[0215] In the embodiment of the present application, on the one hand, by using features between different modal data for replacement, the information of different modal data can be efficiently fused, so that the obtained data features have the characteristics of multi-modal data, improve the expression ability of data features, and further make the result of the target task based on the data features more accurate. On the other hand, by using the sparse scoring network, the scores of some features are greatly different from those of other features, so as to determine which features are useful or useless. On the other hand, the position of the replacement feature is determined by the residual position coding, so as to ensure that the position of the replacement feature in the original feature set is not changed.

[0216] In order to more intuitively see the beneficial effects of the data processing method (hereinafter referred to as Mix) provided by the embodiment of the present application, the performance results of different methods on data set one and data set two are described below.

[0217] The test results are shown in Table 1:

[0218] Table 1

[0219]

[0220] The method includes a Fully Convolutional Networks (FCN), a RefineNet, a FuseNet, a self-supervised model adaptation (SSMA), a cross-Iteration batch normalization (CBN), an RGB-D fusion network (RDFNet), a channel exchanging network (CEN), a fusion method, an input concatenation method, and a MIX method provided by an embodiment of the present application. 32S in FCN-32S indicates that the feature map of the convolutional layer is 1 / 32 of the original image. w / o indicates a same-structure model without multi-modal fusion. [Ti] indicates a tiny model, and [s] indicates a small model. The small model has more layers, more channels, and the like than the tiny model.

[0221] As can be seen from the data in Table 1, the pixel accuracy, the average accuracy, and the average intersection over union of the MIX method are all improved compared with other methods. It can be shown that the method provided by the embodiment of the present application can improve the expression ability of the data features obtained after feature replacement, and thus the result of computer vision obtained is more accurate.

[0222] The data processing method in the embodiment of the present application is described above, and the data processing device in the embodiment of the present application is described below. Referring to Figure 13 One embodiment of the data processing device in the embodiment of the present application includes:

[0223] The obtaining unit 1301 is configured to obtain first data and second data, the modalities of the first data and the second data being different;

[0224] The obtaining unit 1301 is further configured to obtain a first feature set of the first data and a second feature set of the second data;

[0225] The replacing unit 1302 is configured to replace a first target feature in the first feature set with a second target feature in the second feature set to obtain a third feature set, the second target feature corresponding to the first target feature;

[0226] The obtaining unit 1301 is configured to obtain a data feature based on the third feature set and the second feature set, the data feature being used to implement a computer vision task.

[0227] Optionally, the data processing device can further include a determining unit 1303 configured to determine the second target feature based on the first score set and / or the second score set.

[0228] In this embodiment, the operations performed by each unit in the data processing device are similar to those described in the foregoing Figures 1 to 12 embodiments described in the foregoing

[0229] In this embodiment, the replacement unit 1302 uses the features between different modal data for replacement, which can efficiently fuse the information of different modal data, so that the obtained data features have the characteristics of multi-modal data, and the expression ability of the data features is improved.

[0230] Referring to Figure 14 , another structural diagram of a data processing device is provided. The data processing device can include a processor 1401, a memory 1402, and a communication port 1403. The processor 1401, the memory 1402, and the communication port 1403 are interconnected by lines. The memory 1402 stores program instructions and data.

[0231] The memory 1402 stores the corresponding program instructions and data executed by the data processing device in the foregoing Figures 1 to 12 embodiments.

[0232] The processor 1401 is configured to execute the steps performed by the data processing device in any of the embodiments described in the foregoing Figures 1 to 12 embodiments.

[0233] The communication port 1403 can be used for receiving and sending data, and is configured to execute the steps related to obtaining, sending, and receiving in any of the embodiments described in the foregoing Figures 1 to 12 embodiments.

[0234] In an implementation manner, the data processing device can include more or fewer components, and the present application is only illustrative and not limited. Figure 14

[0235] In the several embodiments provided in the present application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device, or unit, and can be electrical, mechanical, or in other forms.

[0236] ​The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0237] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized by software, hardware, firmware or any combination thereof, in whole or in part.

[0238] When the integrated unit is implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0239] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a way of distinguishing the objects with the same properties in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment containing a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

Claims

1. A data processing method, characterized in that: The method is applied to a multimodal fusion scenario, and the method includes: Acquire first data and second data, where the first data and the second data have different modalities; Acquire a first feature set of the first data and a second feature set of the second data; Replacing the first target feature in the first feature set with a second target feature in the second feature set to obtain a third feature set, wherein the second target feature corresponds to the first target feature, and the correspondence between the second target feature and the first target feature is determined according to at least one of the following: a spatial relationship between the first data and the second data, a semantic relationship between the first data and the second data, a position of the first target feature in the first feature set, and a position of the second target feature in the second feature set; Data features are obtained based on the third feature set and the second feature set, where the data features are used to implement a computer vision task.

2. The method according to claim 1, characterized in that The acquiring of data features based on the third feature set and the second feature set includes: replacing the fourth target feature in the second feature set with the third target feature in the first feature set to obtain a fourth feature set, wherein the third target feature corresponds to the fourth target feature; The data features are obtained based on the third feature set and the fourth feature set.

3. The method according to claim 1 or 2, characterized in that Before replacing the first target feature in the first feature set with the second target feature in the second feature set, the method further includes: Obtaining a first score set for the first feature set, where the first features in the first feature set correspond one-to-one to the first scores in the first score set; Obtaining a second score set for the second feature set, where the second features in the second feature set correspond one-to-one to the second scores in the second score set; The second target feature is determined based on the first score set and / or the second score set.

4. The method according to claim 3, characterized in that The obtaining of a first score set for the first feature set includes: Evaluate each feature in the first feature set based on a scoring network to obtain the first score set, wherein the scoring network is used to evaluate the importance of the features; The obtaining of a second score set for the second feature set includes: Each feature in the second feature set is evaluated based on the scoring network to obtain the second score set.

5. The method according to claim 4, characterized in that The output values ​​of the scoring network obey a sparse distribution.

6. The method according to claim 3, characterized in that The obtaining of a first score set for the first feature set includes: Performing a mathematical operation on each first feature in the first feature set to obtain the first score set, where the mathematical operation is performed on each first feature itself, and the mathematical operation includes a rank operation or a modulus operation; The obtaining of a second score set for the second feature set includes: Perform a mathematical operation on each second feature in the second feature set to obtain the second score set.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of a first feature set of the first data and a second feature set of the second data includes: The first feature set and the second feature set are obtained based on a neural network, where the neural network includes an attention network, a multi-layer perceptron, a pooling layer, or a convolutional layer.

8. The method according to claim 7, characterized in that The obtaining the first feature set and the second feature set based on a neural network includes: Splitting the first data to obtain a plurality of first sub-data; Splitting the second data to obtain a plurality of second sub-data; The plurality of first sub-data and the second sub-data are input into the neural network to obtain the first feature set and the second feature set.

9. The method according to any one of claims 1 to 8, characterized in that The replacing the first target feature in the first feature set with the second target feature in the second feature set includes: The first target feature is replaced by the second target feature based on residual position coding, where the residual position coding is used to determine the position of each feature in the first feature set and the second feature set.

10. The method according to any one of claims 1 to 9, characterized in that The neural network further includes a first network layer, and a structure of the first network layer is related to the neural network.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The data features are input into a second network layer to obtain a result of the computer vision task, where the second network layer is related to the computer vision task.

12. The method according to claim 11, characterized in that The computer vision task is a classification task, and the second network layer is a fully connected layer; or the computer vision task is a segmentation task or a detection task, and the second network layer is a convolutional neural network layer or an upsampling layer.

13. A data processing device, characterized in that: The data processing device is applied to a multimodal fusion scenario, and the data processing device includes: an acquiring unit, configured to acquire first data and second data, wherein the first data and the second data have different modalities; The acquisition unit is further configured to acquire a first feature set of the first data and a second feature set of the second data; a replacing unit, configured to replace the first target feature in the first feature set with the second target feature in the second feature set to obtain a third feature set, wherein the second target feature corresponds to the first target feature, and the correspondence between the second target feature and the first target feature is determined according to at least one of the following: a spatial relationship between the first data and the second data, a semantic relationship between the first data and the second data, a position of the first target feature in the first feature set, and a position of the second target feature in the second feature set; The acquisition unit is configured to acquire data features based on the third feature set and the second feature set, where the data features are used to implement a computer vision task.

14. The data processing device according to claim 13, characterized in that The acquiring unit is specifically configured to replace the fourth target feature in the second feature set with the third target feature in the first feature set to obtain a fourth feature set, wherein the third target feature corresponds to the fourth target feature; The acquiring unit is specifically configured to acquire the data features based on the third feature set and the fourth feature set.

15. The data processing device according to claim 13 or 14, characterized in that The acquiring unit is further configured to acquire a first score set of the first feature set, wherein the first features in the first feature set correspond to the first scores in the first score set in a one-to-one correspondence; The acquiring unit is further configured to acquire a second score set of the second feature set, wherein the second features in the second feature set correspond one-to-one to the second scores in the second score set; The data processing device further includes: A determining unit is configured to determine the second target feature based on the first score set and / or the second score set.

16. The data processing device according to claim 15, characterized in that The acquiring unit is specifically configured to evaluate each feature in the first feature set based on a scoring network to obtain the first score set, wherein the scoring network is used to evaluate the importance of the features; The acquisition unit is specifically configured to evaluate each feature in the second feature set based on a scoring network to obtain the second score set.

17. The data processing device according to claim 16, characterized in that The output values ​​of the scoring network obey a sparse distribution.

18. The data processing device according to claim 15, characterized in that The acquiring unit is specifically configured to perform a mathematical operation on each first feature in the first feature set to obtain the first score set, wherein the mathematical operation is performed on each first feature itself, and the mathematical operation includes a rank operation or a modulus operation; The acquisition unit is specifically configured to perform a mathematical operation on each second feature in the second feature set to obtain the second score set.

19. The data processing device according to any one of claims 13 to 18, characterized in that The acquisition unit is specifically used to acquire the first feature set and the second feature set based on a neural network, and the neural network includes an attention network, a multi-layer perceptron, a pooling layer or a convolutional layer.

20. The data processing device according to claim 19, characterized in that The acquiring unit is specifically configured to split the first data into a plurality of first sub-data; The acquiring unit is specifically configured to split the second data into a plurality of second sub-data; The acquisition unit is specifically configured to input the plurality of first sub-data and the second sub-data into the neural network to obtain the first feature set and the second feature set.

21. The data processing device according to any one of claims 13 to 20, characterized in that The replacement unit is specifically configured to replace the first target feature with the second target feature based on residual position coding, where the residual position coding is used to determine the position of each feature in the first feature set and the second feature set.

22. The data processing device according to any one of claims 13 to 21, characterized in that The neural network further includes a first network layer, and a structure of the first network layer is related to the neural network.

23. The data processing device according to any one of claims 13 to 22, characterized in that The acquisition unit is further configured to input the data features into a second network layer to obtain a result of the computer vision task, where the second network layer is related to the computer vision task.

24. The data processing device according to claim 23, characterized in that The computer vision task is a classification task, and the second network layer is a fully connected layer; or the computer vision task is a segmentation task or a detection task, and the second network layer is a convolutional neural network layer or an upsampling layer.

25. A data processing device, characterized in that: include: A processor is coupled to a memory, wherein the memory is used to store a program or an instruction, and when the program or the instruction is executed by the processor, the data processing device executes the method according to any one of claims 1 to 12.

26. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on a terminal device, cause the terminal device to execute the method according to any one of claims 1 to 12.

27. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 12.

Images