Task processing method and related device therefor

Abstract

Disclosed in the present application are a task processing method and a related device therefor, which can relatively accurately execute visual tasks. The method of the present application comprises: first, first data and second data of different modalities for a visual task can be input into a target model; next, the target model can perform feature extraction on the first data and the second data to obtain a first feature and a second feature, the first feature comprising N first sub-features, and the second feature comprising N second sub-features; then, the target model can use the i-th second sub-feature to process the i-th first sub-feature to obtain an i-th third sub-feature, such that a third feature comprising N third sub-features can be obtained; similarly, the target model can further use the i-th first sub-feature to process the i-th second sub-feature to obtain an i-th fourth sub-feature, such that a fourth feature comprising N fourth sub-features can be obtained; and finally, on the basis of the third feature and the fourth feature, the target model can acquire a processing result of the visual task.

Description

A task processing method and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202410508751.8, filed on April 25, 2024, entitled “A Task Processing Method and Related Equipment Thereof”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to artificial intelligence (AI) technology, and more particularly to a task processing method and related equipment. Background Technology

[0003] With the increasing availability of low-cost sensors, multimodal data from various sensors can often be processed comprehensively using neural network models to solve various visual tasks, such as object detection, semantic segmentation, video summarization, and visual question answering. This approach offers significantly better task processing performance than single-modal data.

[0004] In related technologies, when processing a visual task, two types of data representing different modalities for that task can be acquired. These two types of data are then input into a neural network model, which extracts the features from each data set. Next, the neural network model concatenates these two features from different modalities to obtain a concatenated feature. Finally, the neural network model processes this concatenated feature to obtain the result of the visual task.

[0005] In the above process, since the neural network model simply splices the features of different modalities, the processing results of the visual task based on the spliced ​​features are often inaccurate and thus cannot accurately complete the visual task. Summary of the Invention

[0006] This application provides a task processing method and related equipment, which obtains visual task processing results with high accuracy and can complete visual tasks relatively accurately.

[0007] A first aspect of this application provides a task processing method, the method comprising:

[0008] When a visual task exists, first data and second data associated with the visual task can be acquired, wherein the first data consists of data from different modalities. After obtaining the first and second data, they can be input into the target model.

[0009] After obtaining the first data and the second data, the target model can extract features from the first data to obtain the first feature, and extract features from the second data to obtain the second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features, where N is a positive integer greater than or equal to 2.

[0010] For the i-th first sub-feature, the target model can process it using an attention mechanism to obtain the i-th third sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a total of N third sub-features, i.e., the third features. Similarly, for the i-th second sub-feature, the target model can process it using an attention mechanism to obtain the i-th fourth sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a total of N fourth sub-features, i.e., the fourth features.

[0011] After obtaining the third and fourth features, the target model can further process the third and fourth features to obtain the processing results of the visual task.

[0012] As can be seen from the above method, for features of different modalities such as the first feature and the second feature associated with the visual task, since the first feature can be divided into N first sub-features and the second feature can be divided into N second sub-features, the target model can achieve point-to-point fusion of the N first sub-features and the N second sub-features in a one-to-one correspondence (i.e., using the attention mechanism to process the i-th first sub-feature and the i-th second sub-feature). In this way, more factors and details are considered in the feature processing of different modalities. Thus, the processing results of the visual task are often obtained based on the fused features, which are often highly accurate and can complete the visual task more accurately.

[0013] In one possible implementation, processing the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature includes: performing a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; performing a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; performing a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and performing a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature. In the aforementioned implementation, among the N first sub-features included in the first feature, for the i-th first sub-feature, the target model can perform a first calculation on the i-th first sub-feature and a preset Q-weight matrix to obtain the i-th first Q-feature. Then, the target model can also perform a second calculation on the i-th first sub-feature, the i-th second sub-feature, and the preset K-weight matrix to obtain the i-th first K-feature. Then, the target model can perform a third calculation on the i-th second sub-feature and the preset V weight matrix to obtain the i-th first V feature. Subsequently, the target model can perform a fourth calculation on the i-th first Q feature, the i-th first K feature, and the i-th first V feature to obtain the i-th third sub-feature. For the remaining first sub-features other than the i-th first sub-feature, the target model can also perform similar operations, so a total of N third sub-features can be obtained. Thus, it can be seen that when performing multimodal feature fusion, the target model can accurately achieve point-to-point fusion (i.e., the i-th first sub-feature and the i-th second sub-feature) based on the attention mechanism, thereby improving the accuracy of the processing results obtained based on the fused features.

[0014] In one possible implementation, performing a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature includes: performing a second calculation on a preset first perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first K-feature, where the first perturbation value indicates a first interaction degree between the i-th first sub-feature and the i-th second sub-feature; performing a third calculation on the i-th second sub-feature to obtain the i-th first V-feature includes: performing a third calculation on a preset second perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first V-feature, where the second perturbation value indicates a second interaction degree between the i-th first sub-feature and the i-th second sub-feature. In the aforementioned implementation, among the N first sub-features included in the first feature, the target model can perform a first calculation on the i-th first sub-feature and a preset Q-weight matrix to obtain the i-th first Q-feature. Next, the target model can perform a second calculation on the preset first perturbation value, the i-th first sub-feature, the i-th second sub-feature, and the preset K-weight matrix to obtain the i-th first K-feature. Then, the target model can perform a third calculation on the preset second perturbation value, the i-th first sub-feature, the i-th second sub-feature, and the preset V-weight matrix to obtain the i-th first V-feature. Subsequently, the target model can perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature. Therefore, it can be seen that when performing multimodal feature fusion, the target model can accurately achieve point-to-point fusion based on the new attention mechanism. Since the new attention mechanism includes perturbation values, it can adaptively adjust the ratio of self-attention and mutual attention, thereby achieving better multimodal feature fusion and further improving the accuracy of the processing results obtained based on the fused features.

[0015] In one possible implementation, processing the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature includes: performing a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; performing a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; performing a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and performing an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature. In the aforementioned implementation, among the N second sub-features included in the second feature, the target model can perform a fifth calculation on the i-th second sub-feature and a preset Q-weight matrix to obtain the i-th second Q-feature. Then, the target model can also perform a sixth calculation on the i-th second sub-feature, the i-th first sub-feature, and the preset K-weight matrix to obtain the i-th second K-feature. Then, the target model can perform a seventh calculation on the i-th first sub-feature and the preset V weight matrix to obtain the i-th second V feature. Subsequently, the target model can perform an eighth calculation on the i-th second Q feature, the i-th second K feature, and the i-th second V feature to obtain the i-th fourth sub-feature. For the remaining second sub-features other than the i-th second sub-feature, the enhancement layer can also perform similar operations on the remaining second sub-features, so a total of N fourth sub-features can be obtained, i.e., fourth features. It can be seen that when performing multimodal feature fusion, the target model can accurately achieve point-to-point fusion (i.e., the i-th first sub-feature and the i-th second sub-feature) based on the attention mechanism, thereby improving the accuracy of the processing results obtained based on the fused features.

[0016] In one possible implementation, performing a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature includes: performing a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, where the third perturbation value indicates the third interaction degree between the i-th first sub-feature and the i-th second sub-feature; performing a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature includes: performing a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, where the fourth perturbation value indicates the fourth interaction degree between the i-th first sub-feature and the i-th second sub-feature. In the aforementioned implementation, among the N second sub-features included in the second feature, the target model can perform a fifth calculation on the i-th second sub-feature and a preset Q-weight matrix to obtain the i-th second Q-feature. Next, the target model performs a sixth calculation on the preset third perturbation value, the i-th second sub-feature, the i-th first sub-feature, and the preset K-weight matrix to obtain the i-th second K-feature. Then, the target model performs a seventh calculation on the preset fourth perturbation value, the i-th second sub-feature, the i-th first sub-feature, and the preset V-weight matrix to obtain the i-th second V-feature. Subsequently, the target model performs an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature. Thus, it can be seen that when performing multimodal feature fusion, the target model can accurately achieve point-to-point fusion based on the new attention mechanism. Because the new attention mechanism includes perturbation values, it can adaptively adjust the ratio of self-attention and mutual attention, thereby achieving better multimodal feature fusion and further improving the accuracy of the processing results obtained based on the fused features.

[0017] In one possible implementation, obtaining the processing result of the visual task based on the third and fourth features includes: processing the third and fourth features based on an attention mechanism and processing them based on a multilayer perceptron to obtain the processing result of the visual task.

[0018] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0019] A second aspect of this application provides a model training method, comprising: acquiring first data and second data for a visual task, wherein the first data and second data are data of different modalities; processing the first data and second data using a model to be trained to obtain a processing result of the visual task, wherein the model to be trained is used to: extract features from the first data to obtain a first feature, wherein the first feature contains N first sub-features, N≥2; extract features from the second data to obtain a second feature, wherein the second feature contains N second sub-features; perform attention-based processing on the i-th first sub-feature based on the i-th second sub-feature to obtain an i-th third sub-feature, i=1,...,N; perform attention-based processing on the i-th second sub-feature based on the i-th first sub-feature to obtain an i-th fourth sub-feature; obtain the processing result of the visual task based on the third feature and the fourth feature, wherein the third feature contains N third sub-features and the fourth feature contains N fourth sub-features; and train the model to be trained based on the processing result and the actual processing result of the visual task to obtain a target model.

[0020] In one possible implementation, the model to be trained is used to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0021] In one possible implementation, the model to be trained is used to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0022] In one possible implementation, the model to be trained is used to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0023] In one possible implementation, the model to be trained is used to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, where the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, where the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0024] In one possible implementation, the model to be trained is used to: process the third and fourth features based on an attention mechanism and to process them based on a multilayer perceptron, to obtain the processing results of the visual task.

[0025] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0026] A third aspect of this application provides a task processing apparatus comprising a target model. The apparatus includes: an acquisition module for acquiring first data and second data for a visual task, wherein the first data and the second data are data of different modalities; a first extraction module for extracting features from the first data to obtain a first feature, wherein the first feature includes N first sub-features, where N≥2; a second extraction module for extracting features from the second data to obtain a second feature, wherein the second feature includes N second sub-features; a first processing module for performing attention-based processing on the i-th first sub-feature based on the i-th second sub-feature to obtain an i-th third sub-feature, where i=1,...,N; a second processing module for performing attention-based processing on the i-th second sub-feature based on the i-th first sub-feature to obtain an i-th fourth sub-feature; and a third processing module for acquiring the processing result of the visual task based on the third feature and the fourth feature, wherein the third feature includes N third sub-features and the fourth feature includes N fourth sub-features.

[0027] In one possible implementation, the first processing module is configured to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0028] In one possible implementation, the first processing module is configured to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0029] In one possible implementation, the second processing module is configured to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0030] In one possible implementation, the second processing module is configured to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, wherein the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, wherein the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0031] In one possible implementation, a third processing module is used to process the third and fourth features based on an attention mechanism and based on a multilayer perceptron to obtain the processing result of the visual task.

[0032] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0033] A fourth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring first data and second data for a visual task, wherein the first data and the second data are data of different modalities; a processing module for processing the first data and the second data using a model to be trained to obtain a processing result of the visual task, wherein the model to be trained is configured to: extract features from the first data to obtain a first feature, wherein the first feature contains N first sub-features, N≥2; extract features from the second data to obtain a second feature, wherein the second feature contains N second sub-features; perform attention-based processing on the i-th first sub-feature based on the i-th second sub-feature to obtain an i-th third sub-feature, i=1,...,N; perform attention-based processing on the i-th second sub-feature based on the i-th first sub-feature to obtain an i-th fourth sub-feature; acquire the processing result of the visual task based on the third feature and the fourth feature, wherein the third feature contains N third sub-features and the fourth feature contains N fourth sub-features; and a training module for training the model to be trained based on the processing result and the actual processing result of the visual task to obtain a target model.

[0034] In one possible implementation, the model to be trained is used to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0035] In one possible implementation, the model to be trained is used to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0036] In one possible implementation, the model to be trained is used to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0037] In one possible implementation, the model to be trained is used to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, where the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, where the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0038] In one possible implementation, the model to be trained is used to: process the third and fourth features based on an attention mechanism and to process them based on a multilayer perceptron, to obtain the processing results of the visual task.

[0039] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0040] A fifth aspect of this application provides a task processing apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the task processing apparatus performs the method described in the first aspect or any possible implementation thereof.

[0041] A sixth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0042] A seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0043] An eighth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0044] In one possible implementation, the processor is coupled to the memory via an interface.

[0045] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0046] A ninth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0047] A tenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0048] In this embodiment, when processing a visual task, first and second data associated with the visual task can be input into the target model. The first and second data are data from different modalities. Next, the target model can extract features from the first and second data respectively, thereby obtaining a first feature and a second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features. Then, the target model can use the i-th second sub-feature to process the i-th first sub-feature using an attention mechanism, thereby obtaining the i-th third sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a third feature containing N third sub-features. Similarly, the target model can also use the i-th first sub-feature to process the i-th second sub-feature using an attention mechanism, thereby obtaining the i-th fourth sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a fourth feature containing N fourth sub-features. Finally, the target model can obtain the processing result of the visual task based on the third and fourth features. In the aforementioned process, for the first and second features associated with the visual task, which are from different modalities, since the first feature can be divided into N first sub-features and the second feature can be divided into N second sub-features, the target model can achieve point-to-point fusion of the N first sub-features and the N second sub-features in a one-to-one correspondence (i.e., using an attention mechanism to process the i-th first sub-feature and the i-th second sub-feature). This allows for consideration of more factors and details in the feature processing of different modalities. Thus, obtaining the processing results of the visual task based on the fused features often has high accuracy and can complete the visual task more accurately. Attached Figure Description

[0049] Figure 1 is a schematic diagram of a structural framework for artificial intelligence.

[0050] Figure 2a is a schematic diagram of a task processing system provided in an embodiment of this application;

[0051] Figure 2b is another structural schematic diagram of the task processing system provided in an embodiment of this application;

[0052] Figure 2c is a schematic diagram of a task processing device provided in an embodiment of this application;

[0053] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of this application;

[0054] Figure 4 is a flowchart illustrating a task processing method provided in an embodiment of this application.

[0055] Figure 5 is a structural schematic diagram of the target model provided in an embodiment of this application;

[0056] Figure 6 is another structural schematic diagram of the target model provided in the embodiment of this application;

[0057] Figure 7 is a flowchart illustrating a model training method provided in an embodiment of this application.

[0058] Figure 8 is a schematic diagram of a task processing device provided in an embodiment of this application;

[0059] Figure 9 is a schematic diagram of a model training device provided in an embodiment of this application;

[0060] Figure 10 is a structural schematic diagram of an execution device provided in an embodiment of this application;

[0061] Figure 11 is a structural schematic diagram of a training device provided in an embodiment of this application;

[0062] Figure 12 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0063] This application provides a task processing method and related equipment, which obtains visual task processing results with high accuracy and can complete visual tasks relatively accurately.

[0064] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0065] With the increasing availability of low-cost sensors, multimodal data from various sensors can often be processed comprehensively using neural network models to solve various visual tasks, such as object detection, semantic segmentation, video summarization, and visual question answering. This approach offers significantly better task processing performance than single-modal data.

[0066] In related technologies, when processing a visual task, two types of data representing different modalities of that task can be acquired. These two types of data are then input into a neural network model, which extracts the features of each data type. Next, the neural network model concatenates these two features from different modalities to obtain a concatenated feature. Finally, the neural network model processes the concatenated feature to obtain the processing result for the visual task. For example, when processing an autonomous driving task, an image and point cloud for that task can be acquired and input into a neural network model. The neural network model then processes the image and point cloud to obtain the image features and point cloud features, respectively. These features are then concatenated to obtain a concatenated feature, which is further processed to obtain the processing result for the autonomous driving task.

[0067] In the above process, since the neural network model simply splices the features of different modalities, the processing results of the visual task based on the spliced ​​features are often inaccurate and thus cannot accurately complete the visual task.

[0068] To address the aforementioned problems, this application provides a task processing method that can be implemented using artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0069] First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The following explanation of this framework will focus on two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0070] (1) Infrastructure

[0071] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0072] (2) Data

[0073] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0074] (3) Data processing

[0075] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0076] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0077] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0078] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0079] (4) General ability

[0080] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0081] (5) Smart Products and Industry Applications

[0082] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0083] The following sections will introduce several application scenarios for this application.

[0084] Figure 2a is a schematic diagram of a task processing system provided in an embodiment of this application. The task processing system includes user equipment and data processing equipment. The user equipment includes smart terminals such as a mobile phone used by the user and an onboard computer in a vehicle driven by the user. The user equipment is the initiator of task processing; as the initiator of task processing requests, requests are typically initiated by the user through the user equipment.

[0085] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives task processing requests from smart terminals through an interactive interface, and then performs task processing such as machine learning, deep learning, search, reasoning, and decision-making through a storage device for storing data and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0086] In the task processing system shown in Figure 2a, the user device can acquire multiple data points of different modalities collected for a visual task, and then send a request to the data processing device. This prompts the data processing device to perform task processing on the multiple data points of different modalities, thereby obtaining the processing result of the visual task. For example, when the user device has a visual task, it can acquire multiple data points of different modalities associated with the visual task. The user device can then send a task processing request to the data processing device, causing the data processing device to perform a series of processes on the multiple data points of different modalities associated with the target task based on the task processing request, thereby obtaining the processing result of the visual task. For instance, when the visual task is object detection in a scene, the processing result could be the various objects detected in the scene; similarly, when the visual task is semantic segmentation in a scene, the processing result could be the categories of various objects in the scene, and so on.

[0087] In Figure 2a, the data processing device can execute the task processing method of the embodiments of this application.

[0088] Figure 2b is another structural schematic diagram of the task processing system provided in the embodiment of this application. In Figure 2b, the user equipment directly acts as a data processing device. The user equipment can directly obtain input from the user and process it directly by the hardware of the user equipment itself. The specific process is similar to that in Figure 2a, and can be referred to the above description, which will not be repeated here.

[0089] In the task processing system shown in Figure 2b, when the user device has a visual task, it can acquire multiple data points of different modalities associated with the visual task. The user device can then perform a series of processing steps on these multiple data points to obtain the processing result of the visual task. For example, when the visual task is object detection in a scene, the processing result can be the detected objects in that scene. Similarly, when the visual task is semantic segmentation in a scene, the processing result can be the categories of the objects in that scene, and so on.

[0090] In Figure 2b, the user equipment itself can execute the task processing method of the embodiments of this application.

[0091] Figure 2c is a schematic diagram of a task processing device provided in an embodiment of this application.

[0092] The user equipment in Figures 2a and 2b can be local device 301 or local device 302 in Figure 2c. The data processing device in Figure 2a can be execution device 210 in Figure 2c. The data storage system 250 can store the data to be processed by the execution device 210. The data storage system 250 can be integrated on the execution device 210 or set up on the cloud or other network servers.

[0093] The processors in Figures 2a and 2b can be trained on data using neural network models or other models (e.g., support vector machine-based models) for machine learning / deep learning, and then use the trained or learned models to perform image processing tasks to obtain the corresponding processing results.

[0094] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of this application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment of the application may include: various tasks to be scheduled, callable resources, and other parameters.

[0095] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0096] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0097] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0098] In the scenario shown in Figure 3, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0099] It is worth noting that Figure 3 is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in Figure 3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in Figure 3, a neural network can be trained using the training device 120.

[0100] This application also provides a chip including a neural network processor (NPU). This chip can be installed in the execution device 110 shown in FIG. 3 to perform the computational work of the computing module 111. This chip can also be installed in the training device 120 shown in FIG. 3 to perform the training work of the training device 120 and output the target model / rules.

[0101] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0102] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0103] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0104] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0105] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0106] The unified memory is used to store input data and output data.

[0107] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0108] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0109] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0110] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0111] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0112] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0113] (1) Neural Network

[0114] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0115] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0116] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0117] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0118] (2) Backpropagation algorithm

[0119] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0120] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0121] The model training method provided in this application involves the processing of data sequences and can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., the first and second data for the visual task in the model training method provided in this application), ultimately obtaining a trained neural network (e.g., the target model in the model training method provided in this application). Furthermore, the task processing method provided in this application can utilize the trained neural network to input input data (e.g., the first and second data for the visual task in the task processing method provided in this application) into the trained neural network to obtain output data (e.g., the processing result of the visual task in the task processing method provided in this application). It should be noted that the model training method and point cloud acquisition method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0122] It should be noted that the task processing method provided in this application embodiment can be used to process visual tasks in multiple fields. For example, this method can be used to process target detection tasks and semantic segmentation tasks in the field of autonomous driving, so that vehicles can achieve autonomous driving by combining the processing results of target detection tasks and semantic segmentation tasks. As another example, this method can be used to process target detection tasks in the field of mining and smelting, so that ore detection devices can determine the particle size of multiple collected ore particles based on the processing results of target detection tasks. Furthermore, this method can also be used to process visual tasks in the field of logistics and transportation and other fields, which will not be elaborated here. To understand the flow of the task processing method provided in this application embodiment, the flow is described below with reference to Figure 4. Figure 4 is a schematic flowchart of the task processing method provided in this application embodiment. As shown in Figure 4, the method includes:

[0123] 401. Obtain first and second data for the vision task, where the first and second data are data from different modalities.

[0124] In this embodiment, when a visual task exists to be processed, first data and second data associated with the visual task can be acquired. It should be noted that the first data is data of a first modality acquired for the visual task, and the second data is data of a second modality acquired for the visual task. The first and second modalities are different modalities; that is, the first data and the second data are data of different modalities acquired for the same scene. For example, the first data may be an image of a scene captured by a camera, and the second data may be a point cloud obtained by sampling the scene using a LiDAR. Another example is that the first data may be a red / green / blue (RGB) image of a scene captured by a conventional camera, and the second data may be a depth image of the scene captured by a depth camera. Yet another example is that the first data may be an image of a scene captured by a camera, and the second data may be text describing the scene, etc.

[0125] After obtaining the first data and the second data, the first data and the second data can be input into the target model, where the target model is a trained neural network model.

[0126] 402. Perform feature extraction on the first data to obtain the first feature, and perform feature extraction on the second data to obtain the second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features, where N≥2.

[0127] After obtaining the first data and the second data, the target model can extract features from the first data to obtain the first feature, and extract features from the second data to obtain the second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features, where N is a positive integer greater than or equal to 2.

[0128] Specifically, as shown in Figure 5 (Figure 5 is a structural schematic diagram of the target model provided in an embodiment of this application), the target model includes an alignment layer, a feature extraction layer, at least one fusion layer, and a prediction layer. The alignment layer is used to align data from different modalities. The feature extraction layer is used to extract features from the aligned data of different modalities, thereby obtaining features of different modalities. A fusion layer includes an enhancement layer and a point-to-point interaction layer. The enhancement layer is used to enhance the features of different modalities (e.g., based on conventional attention mechanisms), thereby obtaining enhanced features of different modalities. The point-to-point fusion layer is used to interact with the enhanced features of different modalities (e.g., based on novel attention mechanisms), thereby obtaining interactive features. It can be seen that at least one fusion layer continuously enhances and interacts to obtain the final features. The prediction layer is used to make predictions based on the final features, thereby obtaining the processing result of the visual task.

[0129] More specifically, the target model can obtain the first data and the second data in the following ways:

[0130] (1) Since the first data and the second data are data of different modalities, after the first data and the second data are input into the target model, since the first data can contain N first sub-data and the second data can contain N second sub-data, the alignment layer of the target model can align the first data and the second data, that is, align the N first sub-data to the N second sub-data, so as to obtain the aligned first data and the aligned second data.

[0131] For example, as shown in Figure 6 (Figure 6 is another structural schematic diagram of the target model provided in the embodiment of this application), when a certain visual task needs to be processed, an RGB image and a depth image for the visual task can be obtained, and the RGB image and the depth image can be input into the target model. The alignment layer of the target model can first align N pixels of the RGB image with N pixels of the depth image, thereby obtaining the aligned RGB image and depth image.

[0132] (2) After obtaining the aligned first data and the aligned second data, the feature extraction layer of the target model can perform feature extraction on the aligned first data (e.g., convolution, fully connected layers, etc.) to obtain the first feature, and perform feature extraction on the aligned second data to obtain the second feature. It should be noted that the first feature contains N first sub-features corresponding one-to-one with the N first sub-data, and the second feature contains N second sub-features corresponding one-to-one with the N second sub-data.

[0133] Continuing with the example above, after obtaining the aligned RGB image and depth image, since the RGB image contains N pixels, the feature extraction layer of the target model can extract features from these N pixels, thus obtaining the features of these N pixels, i.e., the features of the RGB image. Similarly, since the depth image contains N pixels, the feature extraction layer of the target model can extract features from these N pixels, i.e., the features of the depth image.

[0134] 403. Based on the i-th second sub-feature, process the i-th first sub-feature using an attention mechanism to obtain the i-th third sub-feature, and based on the i-th first sub-feature, process the i-th second sub-feature using an attention mechanism to obtain the i-th fourth sub-feature, i = 1, ..., N.

[0135] After obtaining the first and second features, since the first feature contains N first sub-features, for the i-th first sub-feature (i = 1, ..., N), the target model can perform attention-based processing on the i-th first sub-feature to obtain the i-th third sub-feature. Correspondingly, for the remaining first sub-features other than the i-th first sub-feature, the target model can also perform similar operations on the remaining first sub-features. Therefore, in the end, a total of N third sub-features, corresponding one-to-one with the N first sub-features, can be obtained, which are the third features.

[0136] Similarly, since the second feature contains N second sub-features, for the i-th second sub-feature, the target model can process the i-th second sub-feature using an attention mechanism to obtain the i-th fourth sub-feature. Correspondingly, for the remaining second sub-features other than the i-th second sub-feature, the target model can also perform similar operations on the remaining second sub-features. Therefore, in the end, a total of N fourth sub-features, corresponding one-to-one with the N second sub-features, can be obtained, which are the fourth features.

[0137] Specifically, the target model can obtain the third and fourth features in the following ways:

[0138] (1) Among the N first sub-features contained in the first feature, for the i-th first sub-feature, the enhancement layer of the target model can enhance the i-th first sub-feature (e.g., processing based on multi-head attention mechanism, etc.) to obtain the enhanced i-th first sub-feature. For the other first sub-features besides the i-th first sub-feature, the enhancement layer can also perform similar operations on the other first sub-features, so in the end, a total of N enhanced first sub-features can be obtained, that is, the enhanced first feature.

[0139] (2) Similarly, among the N second sub-features contained in the second feature, for the i-th second sub-feature, the enhancement layer of the target model can enhance the i-th second sub-feature (e.g., processing based on multi-head attention mechanism, etc.) to obtain the enhanced i-th second sub-feature. For the other second sub-features besides the i-th second sub-feature, the enhancement layer can also perform similar operations on the other second sub-features, so in the end, a total of N enhanced second sub-features can be obtained, that is, the enhanced second feature.

[0140] (3) Among the N enhanced first sub-features contained in the enhanced first feature, the point-to-point fusion layer of the target model can perform a first calculation on the enhanced i-th first sub-feature and the preset Q weight matrix to obtain the i-th first Q feature. Next, the point-to-point fusion layer can also perform a second calculation on the enhanced i-th first sub-feature, the enhanced i-th second sub-feature, and the preset K weight matrix to obtain the i-th first K feature. Then, the point-to-point fusion layer can also perform a third calculation on the enhanced i-th second sub-feature and the preset V weight matrix to obtain the i-th first V feature. Subsequently, the point-to-point fusion layer can also perform a fourth calculation on the i-th first Q feature, the i-th first K feature, and the i-th first V feature to obtain the i-th third sub-feature.

[0141] For the enhanced first sub-features other than the enhanced i-th first sub-feature, the enhancement layer can also perform similar operations on the enhanced first sub-features, so a total of N third sub-features can be obtained in the end, that is, the third features.

[0142] (4) Among the enhanced N sub-features contained in the enhanced second feature, the point-to-point fusion layer of the target model can perform a fifth calculation on the enhanced i-th second sub-feature and the preset Q-weight matrix to obtain the i-th second Q-feature. Next, the point-to-point fusion layer can also perform a sixth calculation on the enhanced i-th second sub-feature, the enhanced i-th first sub-feature, and the preset K-weight matrix to obtain the i-th second K-feature. Then, the point-to-point fusion layer can also perform a seventh calculation on the enhanced i-th first sub-feature and the preset V-weight matrix to obtain the i-th second V-feature. Subsequently, the point-to-point fusion layer can also perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0143] For the enhanced second sub-features other than the enhanced i-th second sub-feature, the enhancement layer can also perform similar operations on the enhanced second sub-features, so a total of N fourth sub-features can be obtained in the end, that is, the fourth features.

[0144] Continuing with the example above, the enhancement layer of the target model can enhance the features of the RGB image and the depth image separately to obtain the enhanced features of the RGB image and the enhanced features of the depth image. Since the enhanced RGB image features contain the enhanced features of N pixels of the RGB image, and the enhanced depth image features contain the enhanced features of N pixels of the depth image, for the enhanced features of the i-th pixel of the RGB image... The point-to-point fusion layer can be calculated using the following formula:

[0145] In the above formula, W represents the feature of the i-th pixel in the enhanced RGB image. Q W K and W V These are the preset Q-weight matrix, K-weight matrix, and V-weight matrix, respectively. as well as These are the i-th Q feature, i-th K feature, and i-th V feature of the RGB image, respectively. Let Φ be the fusion feature of the i-th pixel after enhancement of the RGB image. A similar operation can be performed on the features of the remaining pixels after enhancement of the RGB image by the point-to-point fusion layer, thus ultimately obtaining the fusion features of N pixels in the RGB image, i.e., the fusion features of the RGB image. Φ is the model relation discriminator, which can be considered as a preset operation.

[0146] Similarly, the point-to-point fusion layer can perform a similar operation on the enhanced features of N pixels in the depth image, so the final result is the fused features of N pixels in the depth image, which will not be elaborated here.

[0147] More specifically, the target model can also obtain the i-th third sub-feature and the i-th fourth sub-feature in the following ways:

[0148] In the enhanced first feature comprising N enhanced first sub-features, the point-to-point fusion layer of the target model performs a first calculation on the enhanced i-th first sub-feature and a preset Q-weight matrix to obtain the i-th first Q-feature. Next, the point-to-point fusion layer performs a second calculation on a preset first perturbation value, the enhanced i-th first sub-feature, the enhanced i-th second sub-feature, and a preset K-weight matrix to obtain the i-th first K-feature. Then, the point-to-point fusion layer performs a third calculation on a preset second perturbation value, the enhanced i-th first sub-feature, the enhanced i-th second sub-feature, and a preset V-weight matrix to obtain the i-th first V-feature. Finally, the point-to-point fusion layer performs a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0149] It is worth noting that the first perturbation value indicates the first level of interaction (mutual attention) between the i-th first sub-feature and the i-th second sub-feature, and the fifth level of interaction (self-attention) between the i-th first sub-feature and itself. The second perturbation value indicates the second level of interaction (mutual attention) between the i-th first sub-feature and the i-th second sub-feature, and the sixth level of interaction (self-attention) between the i-th second sub-feature and itself. In other words, these two perturbation values ​​can adjust the ratio between mutual attention and self-attention, but the adjustment magnitudes are usually different.

[0150] Similarly, among the N enhanced second sub-features contained in the enhanced second feature, the point-to-point fusion layer of the target model can perform a fifth calculation on the enhanced i-th second sub-feature and the preset Q-weight matrix to obtain the i-th second Q-feature. Next, the point-to-point fusion layer can perform a sixth calculation on the preset third perturbation value, the enhanced i-th second sub-feature, the enhanced i-th first sub-feature, and the preset K-weight matrix to obtain the i-th second K-feature. Then, the point-to-point fusion layer can perform a seventh calculation on the preset fourth perturbation value, the enhanced i-th second sub-feature, the enhanced i-th first sub-feature, and the preset V-weight matrix to obtain the i-th second V-feature. Subsequently, the point-to-point fusion layer can perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0151] It is worth noting that the third perturbation value is used to indicate the third level of interaction (mutual attention) between the i-th first sub-feature and the i-th second sub-feature, and the seventh level of interaction (self-attention) between the i-th second sub-feature and itself. The fourth perturbation value is used to indicate the fourth level of interaction (mutual attention) between the i-th first sub-feature and the i-th second sub-feature, and the sixth level of interaction (self-attention) between the i-th first sub-feature and itself. That is, these two perturbation values ​​can also adjust the ratio between mutual attention and self-attention, but the adjustment magnitudes are usually different.

[0152] Continuing with the example above, for the enhanced features of the i-th pixel in the RGB image... The point-to-point fusion layer can be calculated using the following formula:

[0153] In the above formula, as well as Let L be the perturbation value of the Lth combined layer in the target model. Since the target model contains at least one combined layer, and each combined layer contains an enhancement layer and a point-to-point fusion layer, and the target model in the example shown in Figure 6 may contain multiple combined layers, but here we will use the first combined layer (i.e., the enhancement layer and point-to-point fusion layer in the previous example as the first enhancement layer and the first point-to-point fusion layer) for illustrative purposes, so L = 1. It is worth noting that the perturbation values ​​in different combined layers are usually different. Similarly, for the features of the i-th pixel after enhancement of the depth image, the point-to-point fusion layer can also perform a similar operation, which will not be elaborated here.

[0154] 404. Obtain the processing results of the visual task based on the third feature and the fourth feature. The third feature contains N third sub-features, and the fourth feature contains N fourth sub-features.

[0155] After obtaining the third and fourth features, the target model can further process the third and fourth features to obtain the processing results of the visual task.

[0156] Specifically, the target model can obtain the processing results of the visual task in the following ways:

[0157] When the target model contains multiple ensemble layers, the first ensemble layer obtains the third and fourth features and sends them to the second ensemble layer. The second ensemble layer then performs the same operations on the third and fourth features as the first ensemble layer did on the first and second features (enhancing and interactive operations, i.e., operations based on conventional attention mechanisms and operations based on novel attention mechanisms), and so on, until the last ensemble layer outputs the final two features. The prediction layer of the target model then processes these final two features (e.g., based on a multilayer perceptron) to obtain the result of the visual task.

[0158] As in the example above, in the target model, after the first fusion layer obtains the fusion features of the RGB image and the fusion features of the depth image, the second fusion layer (which includes the second enhancement layer and the second point-to-point fusion layer) can enhance and interact with the fusion features of the RGB image and the fusion features of the depth image. This process continues until the last fusion layer provides the final fusion features of the RGB image and the final fusion features of the depth image to the prediction layer. After the prediction layer processes these features, the result of the visual task can be obtained.

[0159] In addition, the target model provided in the embodiments of this application can be compared with the models provided by some related technologies (methods 1 to 7), and the comparison results are shown in Table 1:

[0160] Table 1

[0161] As shown in Table 1, the performance (evaluation metrics) of the embodiments of this application and related technologies on datasets 1, 2 and 3 shows that the target models provided by the embodiments of this application all have superior performance.

[0162] Furthermore, since the target model provided in this embodiment has the function of adaptively adjusting the ratio of self-attention to mutual attention, the embodiments of this application can also be compared with another part of related technologies. The comparison results are shown in Table 2:

[0163] Table 2

[0164] As shown in Table 1, the performance (evaluation metrics) of the embodiments of this application and related technologies on Dataset 1 shows that the target models provided by the embodiments of this application all have superior performance.

[0165] In this embodiment, when processing a visual task, first and second data associated with the visual task can be input into the target model. The first and second data are data from different modalities. Next, the target model can extract features from the first and second data respectively, thereby obtaining a first feature and a second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features. Then, the target model can use the i-th second sub-feature to process the i-th first sub-feature using an attention mechanism, thereby obtaining the i-th third sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a third feature containing N third sub-features. Similarly, the target model can also use the i-th first sub-feature to process the i-th second sub-feature using an attention mechanism, thereby obtaining the i-th fourth sub-feature. Since i = 1, ..., N, the target model can ultimately obtain a fourth feature containing N fourth sub-features. Finally, the target model can obtain the processing result of the visual task based on the third and fourth features. In the aforementioned process, for the first and second features associated with the visual task, which are from different modalities, since the first feature can be divided into N first sub-features and the second feature can be divided into N second sub-features, the target model can achieve point-to-point fusion of the N first sub-features and the N second sub-features in a one-to-one correspondence (i.e., using an attention mechanism to process the i-th first sub-feature and the i-th second sub-feature). This allows for consideration of more factors and details in the feature processing of different modalities. Thus, obtaining the processing results of the visual task based on the fused features often has high accuracy and can complete the visual task more accurately.

[0166] Furthermore, in the embodiments of this application, the target model has an adaptive adjustment mechanism for self-attention and mutual attention (i.e., the target model can use perturbation values ​​to complete the interaction between different sub-features and the interaction between the sub-features themselves), which can adaptively adjust the ratio of self-attention and mutual attention of multimodal features during fusion, thereby achieving a better fusion between features of different modalities, so as to further improve the accuracy of the processing results of visual tasks.

[0167] The above is a detailed description of the task processing method provided in the embodiments of this application. The following will introduce the model training method provided in the embodiments of this application. Figure 7 is a flowchart illustrating the model training method provided in the embodiments of this application. As shown in Figure 7, the method includes:

[0168] 701. Obtain first and second data for the vision task, where the first and second data are data from different modalities.

[0169] In this embodiment, when the model to be trained needs to be trained, a batch of training data can be obtained. This batch of training data includes first data and second data for the vision task, and the first data and second data are data from different modalities. It should be noted that the actual processing results of the vision task for the first data and the second data are known.

[0170] 702. The training model processes the first and second data to obtain the processing result of the visual task. The training model is used to: extract features from the first data to obtain the first feature, which contains N first sub-features, N≥2; extract features from the second data to obtain the second feature, which contains N second sub-features; process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature, i=1,...,N; process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature; and obtain the processing result of the visual task based on the third and fourth features, where the third feature contains N third sub-features and the fourth feature contains N fourth sub-features.

[0171] After obtaining the first and second data, they can be input into the model to be trained to perform the following operations: extract features from the first data to obtain the first feature, which contains N first sub-features, N≥2; extract features from the second data to obtain the second feature, which contains N second sub-features; process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature, i=1,...,N; process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature; and obtain the (prediction) processing result of the visual task based on the third and fourth features, where the third feature contains N third sub-features and the fourth feature contains N fourth sub-features.

[0172] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0173] In one possible implementation, the model to be trained is used to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0174] In one possible implementation, the model to be trained is used to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0175] In one possible implementation, the model to be trained is used to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0176] In one possible implementation, the model to be trained is used to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, where the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, where the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0177] In one possible implementation, the model to be trained is used to process the third and fourth features based on an attention mechanism and based on a multilayer perceptron to obtain the processing results of the visual task.

[0178] For an explanation of step 702, please refer to the relevant descriptions of steps 402 to 404; they will not be repeated here.

[0179] 703. Based on the processing results and the actual processing results of the visual task, train the model to be trained to obtain the target model.

[0180] After obtaining the processing result of the visual task, since the actual processing result of the visual task is known, the target loss can be obtained using both the processing result and the actual processing result. The target loss is used to indicate the difference between the processing result of the visual task and the actual processing result. Then, the parameters of the model to be trained can be updated using the target loss to obtain the updated model to be trained. The updated model can then be trained using the next batch of training data until the model training conditions are met, resulting in the target model shown in the embodiment of Figure 4.

[0181] The target model trained in this embodiment has task processing capabilities. Specifically, when processing a visual task, first and second data associated with the visual task can be input into the target model. The first and second data are data from different modalities. Next, the target model can extract features from the first and second data respectively, thereby obtaining a first feature and a second feature. The first feature contains N first sub-features, and the second feature contains N second sub-features. Then, the target model can use the i-th second sub-feature to process the i-th first sub-feature using an attention mechanism, thereby obtaining the i-th third sub-feature. Since i = 1, ..., N, the target model ultimately obtains a third feature containing N third sub-features. Similarly, the target model can also use the i-th first sub-feature to process the i-th second sub-feature using an attention mechanism, thereby obtaining the i-th fourth sub-feature. Since i = 1, ..., N, the target model ultimately obtains a fourth feature containing N fourth sub-features. Finally, the target model can obtain the processing result of the visual task based on the third and fourth features. In the aforementioned process, for the first feature and the second feature associated with the visual task, which are different modal features, since the first feature can be divided into N first sub-features and the second feature can be divided into N second sub-features, the target model can achieve point-to-point fusion of the N first sub-features and the N second sub-features in a one-to-one correspondence. This allows for consideration of more factors and details in the feature processing of different modalities. Thus, obtaining the processing results of the visual task based on the fused features often has high accuracy and can complete the visual task more accurately.

[0182] The above is a detailed description of the task processing method and model training method provided in the embodiments of this application. The task processing device and model training device will be described below. Figure 8 is a schematic diagram of the structure of a task processing device provided in an embodiment of this application. As shown in Figure 8, the task processing device includes: an acquisition module 801, an extraction module 802, and a processing module 803. The extraction module 802 includes a first extraction module and a second extraction module, and the processing module includes a first processing module, a second processing module, and a third processing module.

[0183] The acquisition module 801 is used to acquire first data and second data for a vision task, wherein the first data and the second data are data of different modalities.

[0184] The first extraction module is used to extract features from the first data to obtain the first feature, which contains N first sub-features, where N≥2;

[0185] The second extraction module is used to extract features from the second data to obtain the second feature, which contains N second sub-features.

[0186] The first processing module is used to process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature, i = 1, ..., N;

[0187] The second processing module is used to process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature;

[0188] The third processing module is used to obtain the processing results of the visual task based on the third feature and the fourth feature. The third feature contains N third sub-features, and the fourth feature contains N fourth sub-features.

[0189] In one possible implementation, the first processing module is configured to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0190] In one possible implementation, the first processing module is configured to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0191] In one possible implementation, the second processing module is configured to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0192] In one possible implementation, the second processing module is configured to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, wherein the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, wherein the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0193] In one possible implementation, a third processing module is used to process the third and fourth features based on an attention mechanism and based on a multilayer perceptron to obtain the processing result of the visual task.

[0194] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0195] Figure 9 is a schematic diagram of a model training device provided in an embodiment of this application. As shown in Figure 9, the model training device includes:

[0196] The acquisition module 901 is used to acquire first data and second data for a visual task, wherein the first data and the second data are data of different modalities.

[0197] Processing module 902 is used to process the first data and the second data using the model to be trained to obtain the processing result of the visual task. The model to be trained is used to: extract features from the first data to obtain a first feature, which contains N first sub-features, N≥2; extract features from the second data to obtain a second feature, which contains N second sub-features; process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature, i=1,...,N; process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature; and obtain the processing result of the visual task based on the third and fourth features, where the third feature contains N third sub-features and the fourth feature contains N fourth sub-features.

[0198] Training module 903 is used to train the model to be trained based on the processing results and the real processing results of the visual task, so as to obtain the target model.

[0199] In one possible implementation, the model to be trained is used to: perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; perform a second calculation on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; perform a third calculation on the i-th second sub-feature to obtain the i-th first V-feature; and perform a fourth calculation on the i-th first Q-feature, the i-th first K-feature, and the i-th first V-feature to obtain the i-th third sub-feature.

[0200] In one possible implementation, the model to be trained is used to: perform a second calculation on a preset first perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first K-feature, wherein the first perturbation value is used to indicate a first degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a third calculation on a preset second perturbation value, an i-th first sub-feature, and an i-th second sub-feature to obtain an i-th first V-feature, wherein the second perturbation value is used to indicate a second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0201] In one possible implementation, the model to be trained is used to: perform a fifth calculation on the i-th second sub-feature to obtain the i-th second Q-feature; perform a sixth calculation on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; perform a seventh calculation on the i-th first sub-feature to obtain the i-th second V-feature; and perform an eighth calculation on the i-th second Q-feature, the i-th second K-feature, and the i-th second V-feature to obtain the i-th fourth sub-feature.

[0202] In one possible implementation, the model to be trained is used to: perform a sixth calculation on a preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature, where the third perturbation value is used to indicate the third degree of interaction between the i-th first sub-feature and the i-th second sub-feature; and perform a seventh calculation on a preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature, where the fourth perturbation value is used to indicate the fourth degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

[0203] In one possible implementation, the model to be trained is used to: process the third and fourth features based on an attention mechanism and to process them based on a multilayer perceptron, to obtain the processing results of the visual task.

[0204] In one possible implementation, the first data contains N first sub-data, the second data contains N second sub-data, the N first sub-data and the N second sub-data correspond one-to-one, the N first sub-data and the N first sub-features correspond one-to-one, and the N second sub-data and the N second sub-features correspond one-to-one.

[0205] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0206] This application also relates to an execution device. Figure 10 is a schematic diagram of the execution device provided in this application. As shown in Figure 10, the execution device 1000 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. The execution device 1000 may be equipped with the task processing device described in the embodiment corresponding to Figure 8, used to implement the task processing function in the embodiment corresponding to Figure 4. Specifically, the execution device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003, and a memory 1004 (the number of processors 1003 in the execution device 1000 can be one or more; Figure 10 uses one processor as an example). The processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of this application, the receiver 1001, transmitter 1002, processor 1003, and memory 1004 can be connected via a bus or other means.

[0207] Memory 1004 may include read-only memory and random access memory, and provides instructions and data to processor 1003. A portion of memory 1004 may also include non-volatile random access memory (NVRAM). Memory 1004 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0208] Processor 1003 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0209] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1003. The processor 1003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 1003 or by instructions in software form. The processor 1003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1003 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1004. Processor 1003 reads the information in memory 1004 and, in conjunction with its hardware, completes the steps of the above method.

[0210] Receiver 1001 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1002 can be used to output digital or character information through the first interface; transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1002 may also include a display device such as a display screen.

[0211] In one embodiment of this application, the processor 1003 is used to obtain the processing result of the visual task through the target model in the embodiment corresponding to FIG4.

[0212] This application also relates to a training device. Figure 11 is a schematic diagram of the structure of a training device provided in this application embodiment. As shown in Figure 11, the training device 1100 is implemented by one or more servers. The training device 1100 can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1111 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 may be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1111 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the training device 1100.

[0213] The training device 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158; or, one or more operating systems 1141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0214] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 7.

[0215] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0216] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0217] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0218] Specifically, please refer to Figure 12, which is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1200. The NPU 1200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0219] In some implementations, the arithmetic circuit 1203 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1203 is a two-dimensional pulsating array. The arithmetic circuit 1203 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1203 is a general-purpose matrix processor.

[0220] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1202 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1201 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1208.

[0221] Unified memory 1206 is used to store input and output data. Weight data is directly transferred to weight memory 1202 via Direct Memory Access Controller (DMAC) 1205. Input data is also transferred to unified memory 1206 via DMAC.

[0222] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1209.

[0223] The Bus Interface Unit (BIU) 1210 is used by the instruction fetch memory 1209 to fetch instructions from external memory, and also by the memory access controller 1205 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0224] The DMAC is mainly used to move input data from external memory DDR to unified memory 1206, or to weight data to weight memory 1202, or to input data to input memory 1201.

[0225] The vector computation unit 1207 includes multiple processing units that further process the output of the computation circuit 1203 when needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0226] In some implementations, the vector computation unit 1207 can store the processed output vector in the unified memory 1206. For example, the vector computation unit 1207 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1203, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1207 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1203, for example, for use in subsequent layers of the neural network.

[0227] The instruction fetch buffer 1209 connected to the controller 1204 is used to store the instructions used by the controller 1204;

[0228] Unified memory 1206, input memory 1201, weight memory 1202, and instruction fetch memory 1209 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0229] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0230] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0232] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0233] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A task processing method, characterized in that, The method is implemented through a target model, and the method includes: Acquire first and second data for a vision task, wherein the first and second data are data of different modalities; Feature extraction is performed on the first data to obtain a first feature, which contains N first sub-features, where N≥2; Feature extraction is performed on the second data to obtain a second feature, which contains N second sub-features; Based on the i-th second sub-feature, the i-th first sub-feature is processed using an attention mechanism to obtain the i-th third sub-feature, i = 1, ..., N; Based on the i-th first sub-feature, the i-th second sub-feature is processed using an attention mechanism to obtain the i-th fourth sub-feature; The processing result of the visual task is obtained based on the third feature and the fourth feature, wherein the third feature contains N third sub-features and the fourth feature contains N fourth sub-features.

2. The method according to claim 1, characterized in that, The process of processing the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature includes: Perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; A second calculation is performed on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; A third calculation is performed on the i-th second sub-feature to obtain the i-th first V feature; A fourth calculation is performed on the i-th first Q feature, the i-th first K feature, and the i-th first V feature to obtain the i-th third sub-feature.

3. The method according to claim 2, characterized in that, The second calculation performed on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature includes: A second calculation is performed on the preset first perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first K-feature. The first perturbation value is used to indicate the first interaction degree between the i-th first sub-feature and the i-th second sub-feature. The third calculation performed on the i-th second sub-feature to obtain the i-th first V feature includes: A third calculation is performed on the preset second perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first V feature. The second perturbation value is used to indicate the second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

4. The method according to claim 1, characterized in that, The process of processing the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature includes: The fifth calculation is performed on the i-th second sub-feature to obtain the i-th second Q-feature; A sixth calculation is performed on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; The i-th first sub-feature is subjected to a seventh calculation to obtain the i-th second V feature; The i-th second Q feature, the i-th second K feature, and the i-th second V feature are subjected to an eighth calculation to obtain the i-th fourth sub-feature.

5. The method according to claim 4, characterized in that, The sixth calculation performed on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature includes: A sixth calculation is performed on the preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature. The third perturbation value is used to indicate the third interaction degree between the i-th first sub-feature and the i-th second sub-feature. The seventh calculation performed on the i-th first sub-feature to obtain the i-th second V feature includes: A seventh calculation is performed on the preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature. The fourth perturbation value is used to indicate the fourth interaction degree between the i-th first sub-feature and the i-th second sub-feature.

6. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining the visual task result based on the third and fourth features includes: The third and fourth features are processed using an attention-based mechanism and a multilayer perceptron-based process to obtain the processing result of the visual task.

7. The method according to any one of claims 1 to 6, characterized in that, The first data contains N first sub-data, and the second data contains N second sub-data. The N first sub-data correspond one-to-one with the N second sub-data, the N first sub-data correspond one-to-one with the N first sub-features, and the N second sub-data correspond one-to-one with the N second sub-features.

8. A model training method, characterized in that, The method includes: Acquire first and second data for a vision task, wherein the first and second data are data of different modalities; The first data and the second data are processed by a model to be trained to obtain the processing result of the visual task. The model to be trained is used for: extracting features from the first data to obtain a first feature, the first feature containing N first sub-features, N≥2; extracting features from the second data to obtain a second feature, the second feature containing N second sub-features; processing the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain an i-th third sub-feature, i=1,...,N; processing the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain an i-th fourth sub-feature; and obtaining the processing result of the visual task based on the third feature and the fourth feature, the third feature containing N third sub-features and the fourth feature containing N fourth sub-features. Based on the processing results and the actual processing results of the visual task, the model to be trained is trained to obtain the target model.

9. The method according to claim 8, characterized in that, The model to be trained is used for: Perform a first calculation on the i-th first sub-feature to obtain the i-th first Q-feature; A second calculation is performed on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th first K-feature; A third calculation is performed on the i-th second sub-feature to obtain the i-th first V feature; A fourth calculation is performed on the i-th first Q feature, the i-th first K feature, and the i-th first V feature to obtain the i-th third sub-feature.

10. The method according to claim 9, characterized in that, The model to be trained is used for: A second calculation is performed on the preset first perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first K-feature. The first perturbation value is used to indicate the first interaction degree between the i-th first sub-feature and the i-th second sub-feature. A third calculation is performed on the preset second perturbation value, the i-th first sub-feature, and the i-th second sub-feature to obtain the i-th first V feature. The second perturbation value is used to indicate the second degree of interaction between the i-th first sub-feature and the i-th second sub-feature.

11. The method according to claim 8, characterized in that, The model to be trained is used for: The fifth calculation is performed on the i-th second sub-feature to obtain the i-th second Q-feature; A sixth calculation is performed on the i-th second sub-feature and the i-th first sub-feature to obtain the i-th second K-feature; The i-th first sub-feature is subjected to a seventh calculation to obtain the i-th second V feature; The i-th second Q feature, the i-th second K feature, and the i-th second V feature are subjected to an eighth calculation to obtain the i-th fourth sub-feature.

12. The method according to claim 11, characterized in that, The model to be trained is used for: A sixth calculation is performed on the preset third perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second K-feature. The third perturbation value is used to indicate the third interaction degree between the i-th first sub-feature and the i-th second sub-feature. A seventh calculation is performed on the preset fourth perturbation value, the i-th second sub-feature, and the i-th first sub-feature to obtain the i-th second V-feature. The fourth perturbation value is used to indicate the fourth interaction degree between the i-th first sub-feature and the i-th second sub-feature.

13. The method according to any one of claims 8 to 12, characterized in that, The model to be trained is used for: The third and fourth features are processed using an attention-based mechanism and a multilayer perceptron-based process to obtain the processing result of the visual task.

14. The method according to any one of claims 8 to 13, characterized in that, The first data contains N first sub-data, and the second data contains N second sub-data. The N first sub-data correspond one-to-one with the N second sub-data, the N first sub-data correspond one-to-one with the N first sub-features, and the N second sub-data correspond one-to-one with the N second sub-features.

15. A task processing device, characterized in that, The device includes a target model, and the device comprises: The acquisition module is used to acquire first data and second data for a vision task, wherein the first data and the second data are data of different modalities; The first extraction module is used to extract features from the first data to obtain a first feature, wherein the first feature contains N first sub-features, where N≥2; The second extraction module is used to extract features from the second data to obtain a second feature, which contains N second sub-features. The first processing module is used to process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain the i-th third sub-feature, i = 1, ..., N; The second processing module is used to process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain the i-th fourth sub-feature; The third processing module is used to obtain the processing result of the visual task based on the third feature and the fourth feature, wherein the third feature contains N third sub-features and the fourth feature contains N fourth sub-features.

16. A model training device, characterized in that, The device includes: The acquisition module is used to acquire first data and second data for a vision task, wherein the first data and the second data are data of different modalities; The processing module is used to process the first data and the second data using a model to be trained, and obtain the processing result of the visual task. The model to be trained is used to: extract features from the first data to obtain a first feature, the first feature containing N first sub-features, N≥2; extract features from the second data to obtain a second feature, the second feature containing N second sub-features; process the i-th first sub-feature based on the i-th second sub-feature using an attention mechanism to obtain an i-th third sub-feature, i=1,...,N; process the i-th second sub-feature based on the i-th first sub-feature using an attention mechanism to obtain an i-th fourth sub-feature; and obtain the processing result of the visual task based on the third feature and the fourth feature, the third feature containing N third sub-features and the fourth feature containing N fourth sub-features. The training module is used to train the model to be trained based on the processing results and the actual processing results of the visual task, so as to obtain the target model.

17. A task processing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the task processing device performs the method as described in any one of claims 1 to 14.

18. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 14.

19. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 14.

Images