Data processing method and apparatus therefor

Abstract

A data processing method, which is applied to the field of artificial intelligence. In the present application, the operation based on a cross-channel self-attention operation mechanism and the operation based on a cross-space self-attention mechanism are simultaneously performed in the same network layer, and a channel self-attention mechanism is combined with a grid-based spatial self-attention mechanism, such that not only can a direct dependency relationship between different spatial positions be established to enable direct interaction between long-distance pixels with spatial self-attention missing, but the spatial dependence between windows can also be captured.

Description

A data processing method and device thereof

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 27, 2024, with application number 202410216885.2 and application name “A data processing method and device thereof”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. Background Art

[0003] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0004] Transformer-based networks are a type of neural network based on the self-attention mechanism. Existing implementations of transformer-based networks either calculate self-attention along the channel dimension, failing to establish direct dependencies between different spatial locations, or calculate self-attention within the spatial dimension, failing to clearly and completely capture the spatial dependencies between windows, resulting in poor network processing accuracy.

[0005] Therefore, it is urgent to design a network architecture that can capture the spatial dependencies between windows and establish direct dependencies between different spatial locations. Summary of the Invention

[0006] In the first aspect, the present application provides a data processing method, which is applied to the processing of a machine learning model, wherein the machine learning model includes a first network, a target network layer and a second network. The first network layer can be a network connected before the target network layer in the machine learning model, and the second network layer can be a network connected after the target network layer in the machine learning model. When processing data, the machine learning model can obtain a first feature representation of the target data through the first network based on the target data input into the machine learning model, and perform a feature extraction operation through the target network layer based on the first feature representation to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing an operation of a self-attention operation mechanism between different channels and performing an operation of a self-attention mechanism between different spaces, and the target network layer is a network layer in the first network; according to the second feature representation, the processing result of the target data is obtained through the second network.

[0007] In existing implementations, when implementing attention operations, the target network layer either calculates self-attention along the channel dimension, failing to establish a direct dependency between different spatial positions, or calculates self-attention within the spatial dimension, failing to clearly and completely capture the spatial dependencies between windows. In an embodiment of the present application, the self-attention operation mechanism between different channels and the self-attention mechanism between different spaces are simultaneously performed within the same network layer, combining the channel self-attention mechanism with the grid-based spatial self-attention mechanism, thereby establishing a direct dependency between different spatial positions, making up for the direct interaction between distant pixels that is missing in spatial self-attention, and capturing the spatial dependencies between windows.

[0008] The target network layer can be a network layer in the first network, for example, a transformer layer in the first network. Implementing both spatial self-attention and channel self-attention in the same network layer can achieve interactive operations between two dimensions (space and channel). Compared with performing spatial self-attention and channel self-attention in different network layers, the network's ability to learn the direct dependencies between different spatial positions and the spatial dependencies between windows can be further improved. In one possible implementation, the target network layer is a transformer layer.

[0009] For example, the target data is image data, and the machine learning model can be used to perform image processing tasks, and the processing result obtained by processing the target data through the machine learning model is the processing result of the image processing task.

[0010] For example, the target data is audio data, and the machine learning model can be used to perform an audio processing task, and the processing result obtained by processing the target data through the machine learning model is the processing result of the audio processing task.

[0011] For example, the target data is text data, and the machine learning model can be used to perform a text processing task, and the processing result obtained by processing the target data through the machine learning model is the processing result of the text processing task.

[0012] For example, image processing tasks may include image classification, target recognition, image enhancement (such as but not limited to super-resolution, denoising, rain removal, deblurring, demosaicing, etc.), image generation, and the like.

[0013] For example, audio processing tasks can include speech-to-text, audio enhancement, audio synthesis, and the like.

[0014] For example, text processing tasks can be summary generation, dialogue response, text generation and other tasks.

[0015] In one possible implementation, the target network layer includes a first attention module and a second attention module, and the feature extraction operation is performed through the target network layer based on the first feature representation, including: performing a self-attention operation mechanism between different channels on the first feature representation through the first attention module; and performing a self-attention mechanism operation between different spaces on the first feature representation after the self-attention operation mechanism between different channels through the second attention module.

[0016] That is to say, the first attention module in the target network layer for performing the self-attention operation mechanism between different channels can be connected in series with the second attention module for performing the self-attention operation mechanism between different spaces, and the first attention module is connected before the second attention module.

[0017] In one possible implementation, the target network layer includes a first attention module and a second attention module, and the feature extraction operation is performed through the target network layer based on the first feature representation, including: performing a self-attention mechanism operation between different spaces on the first feature representation through the first attention module; and performing a self-attention operation mechanism operation between different channels on the first feature representation after the self-attention operation within the spatial dimension through the second attention module.

[0018] That is to say, the first attention module in the target network layer for performing the self-attention operation mechanism between different channels can be connected in series with the second attention module for performing the self-attention operation mechanism between different spaces, and the second attention module is connected before the first attention module.

[0019] In one possible implementation, the target network layer includes a first processing branch and a second processing branch in parallel, the first processing branch includes a first attention module, the second processing branch includes a second attention module, and the processing result of the first processing branch is used to be fused with the processing result of the second processing branch.

[0020] That is to say, the first attention module in the target network layer used to perform the self-attention operation mechanism between different channels and the second attention module used to perform the self-attention operation mechanism between different spaces may not be connected in series, but belong to different branches in parallel.

[0021] For example, the first attention module and the second attention module can be connected in parallel. That is, the first attention module for performing the self-attention operation mechanism between different channels in the target network layer can be connected in parallel with the second attention module for performing the self-attention operation mechanism between different spaces. The first attention module can be used to perform the self-attention operation mechanism between different channels on the first feature representation; the second attention module can be used to perform the self-attention operation mechanism between different spaces on the first feature representation; and the first feature representation after the self-attention operation mechanism between different channels and the first feature representation after the self-attention operation within the spatial dimension are fused.

[0022] In one possible implementation, the operation of the self-attention operation mechanism between different channels includes: dividing the Q matrix, K matrix and V matrix of the input features respectively to obtain multiple first Q matrices, multiple first K matrices and multiple first V matrices; wherein, the first Q matrix, the first K matrix and the first V matrix correspond one to one, and the sizes of the corresponding first Q matrix, the first K matrix and the first V matrix are consistent; according to the multiple first Q matrices, the multiple first K matrices and the multiple first V matrices, multiple first calculation results are obtained, each of the first calculation results is obtained by operation between the corresponding first Q matrix, the first K matrix and the first V matrix; the fusion result of the multiple first operation results is used to obtain the result of the operation of the self-attention operation mechanism between different channels.

[0023] The channel-wise self-attention mechanism (CSA) computes self-attention along the channel dimension, achieving computational complexity that is linear with respect to resolution. The CSA is a spatially global self-attention computation method that can achieve a certain degree of global perspective. However, this global perspective is indirectly obtained by calculating the channel-wise self-attention matrix using a global scope. This results in a coarse global perspective that cannot accurately capture and establish global dependencies.

[0024] In an embodiment of the present application, in order to make the channel self-attention mechanism pay more attention to local information, the original global-based channel self-attention mechanism is improved to a local context-based channel self-attention mechanism.

[0025] In one possible implementation, the Q matrix, K matrix, and V matrix of the input features can be shuffled and partitioned to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices. The second Q matrices, the second K matrices, and the second V matrices correspond one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size. Unlike Method 1, Method 1 is implemented by directly splitting the partitioned windows and calculating channels within non-overlapping windows. Method 2 first slides the window spatially and then calculates window-channel self-attention in the same manner. Similar to Method 1, Method 2 can also obtain multiple second calculation results based on the multiple second Q matrices, the multiple second K matrices, and the multiple second V matrices. Each second calculation result is obtained by operating on a corresponding second Q matrix, a second K matrix, and a second V matrix. The fusion result (e.g., concatenation) of the multiple second calculation results is used to obtain the result of the self-attention operation mechanism between the different channels.

[0026] In a possible implementation, the method further includes: obtaining the result of the operation of the self-attention operation mechanism between the different channels by fusing the fusion result of multiple first operation results and the fusion result of the multiple second operation results.

[0027] In one possible implementation, the first network, the target network layer or the second network further includes: a feedforward network FFN; the FFN is used to perform multiple parallel depth-separable convolution operations and / or multiple serial point convolution operations on the result obtained by nonlinear activation; or, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation.

[0028] Existing technologies use a local enhancement feedforward network to improve local feature extraction by inserting a 3x3 depthwise separable convolution between 1x1 convolutions. However, the receptive field of 3x3 depthwise separable convolution is limited, making it incapable of extracting larger-scale local features.

[0029] In an embodiment of the present application, multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations are connected after the nonlinear layer, so that local features of a larger scale can be extracted, significantly enhancing the performance of FFN.

[0030] In one possible implementation,

[0031] During model training, the FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation;

[0032] During model inference, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

[0033] During training, the network's representational capabilities are enhanced by adding serial point convolutions and parallel depthwise separable convolutions with various kernel sizes. During inference, the serial pointwise convolutions are reparameterized into a single pointwise convolution, and the parallel multi-way depthwise separable convolutions are reparameterized into a single depthwise separable convolution with a single large kernel. This significantly reduces the computational complexity of the inference phase without sacrificing the model's representational capabilities.

[0034] In a second aspect, the present application provides a data processing device for processing a machine learning model, wherein the machine learning model includes a first network, a target network layer, and a second network, and the device includes:

[0035] Acquisition module, used to obtain target data;

[0036] a processing module, configured to obtain, based on the target data, a first feature representation of the target data through the first network; and, based on the first feature representation, perform a feature extraction operation through the target network layer to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing a self-attention operation mechanism between different channels and performing a self-attention operation mechanism between different spaces, and the target network layer is a network layer in the first network;

[0037] According to the second feature representation, a processing result of the target data is obtained through the second network.

[0038] In one possible implementation,

[0039] The target data is image data, and the processing result is the processing result of the image processing task; or,

[0040] The target data is audio data, and the processing result is the processing result of the audio processing task; or

[0041] The target data is text data, and the processing result is the processing result of the text processing task.

[0042] In a possible implementation, the target network layer includes a first attention module and a second attention module, and the processing module is specifically configured to:

[0043] performing, by the first attention module, a self-attention operation mechanism between different channels on the first feature representation;

[0044] Through the second attention module, the first feature representation after the self-attention operation mechanism between the different channels is operated by the self-attention mechanism between different spaces.

[0045] In a possible implementation, the target network layer includes a first attention module and a second attention module, and the processing module is specifically configured to:

[0046] Performing a self-attention mechanism operation between different spaces on the first feature representation through the first attention module;

[0047] Through the second attention module, the first feature representation after the self-attention operation in the spatial dimension is operated by the self-attention operation mechanism between different channels.

[0048] In one possible implementation, the target network layer includes a first processing branch and a second processing branch in parallel, the first processing branch includes a first attention module, the second processing branch includes a second attention module, and the processing result of the first processing branch is used to be fused with the processing result of the second processing branch.

[0049] In one possible implementation, the operation of the self-attention operation mechanism between different channels includes:

[0050] Dividing the Q matrix, K matrix, and V matrix of the input features respectively to obtain a plurality of first Q matrices, a plurality of first K matrices, and a plurality of first V matrices; wherein the first Q matrices, the first K matrices, and the first V matrices correspond to each other one-to-one, and the corresponding first Q matrices, the first K matrices, and the first V matrices have the same size;

[0051] Obtaining a plurality of first calculation results based on the plurality of first Q matrices, the plurality of first K matrices, and the plurality of first V matrices, each of the first calculation results being obtained by operation between a first Q matrix, a first K matrix, and a first V matrix having a corresponding relationship;

[0052] The multiple first operation results are fused to obtain the operation result of the self-attention operation mechanism between the different channels.

[0053] In one possible implementation, the operation of the self-attention operation mechanism between different channels includes:

[0054] Scrambling and dividing the elements included in the Q matrix, K matrix, and V matrix of the input features respectively to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices; wherein the second Q matrices, the second K matrices, and the second V matrices correspond to each other one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size;

[0055] Based on the multiple second Q matrices, the multiple second K matrices and the multiple second V matrices, multiple second calculation results are obtained, each of the second calculation results is obtained through operation between the second Q matrix, the second K matrix and the second V matrix with a corresponding relationship; the fusion result of the multiple second operation results is used to obtain the operation result of the self-attention operation mechanism between the different channels.

[0056] In a possible implementation, the processing module is further configured to:

[0057] By fusing the fusion results of the multiple first operation results and the fusion results of the multiple second operation results, the operation results of the self-attention operation mechanism between the different channels are obtained.

[0058] In a possible implementation, the first network, the target network layer or the second network further includes: a feed-forward network FFN;

[0059] The FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; or,

[0060] The FFN is used to perform a target point convolution operation on the result obtained by the nonlinear activation.

[0061] In one possible implementation,

[0062] During model training, the FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation;

[0063] During model inference, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

[0064] In a third aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0066] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0067] In a sixth aspect, the present application provides a chip system comprising a processor configured to support the execution of a data processing device to implement the functions described in the aforementioned aspects, such as transmitting or processing the data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory configured to store program instructions and data necessary for executing the device or training the device. The chip system may consist of a single chip or may include a chip and other discrete components. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0069] FIG2 is a schematic diagram of the structure of a server;

[0070] FIG3 is a flowchart of a data processing method provided in an embodiment of the present application;

[0071] Figure 4 shows a process of cloud services;

[0072] FIG5 is a flowchart of a data processing method provided in an embodiment of the present application;

[0073] FIG6 is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0074] FIG7 is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0075] FIG8 is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0076] FIG9A is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0077] FIG9B is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0078] FIG9C is a schematic diagram of an effect provided by an embodiment of the present application;

[0079] FIG10 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0080] FIG11 is a schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0081] FIG12 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

[0082] FIG13 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0084] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0085] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0086] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent variations in measurements or calculations that one of ordinary skill in the art would recognize. Furthermore, the use of "may" when describing embodiments of the present invention refers to "one or more possible embodiments." As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0087] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0088] (1) Infrastructure

[0089] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0090] (2) Data

[0091] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0092] (3) Data processing

[0093] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0094] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0095] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0096] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0097] (4) General ability

[0098] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0099] (5) Smart products and industry applications

[0100] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0101] It should be understood that the steps related to the model reasoning process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture described above. The system architecture provided by the embodiments of this application is described in detail below with reference to Figure 2.

[0102] FIG2 is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in FIG2 , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition system 560 .

[0103] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The computing module 511 may include the machine learning model / rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0104] The execution device 510 may be a terminal device or a server that runs the above-mentioned image editing and recognition application.

[0105] The data acquisition device 560 is used to collect training samples. The training samples can be image data, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0106] The training device 520 can train the neural network (such as the machine learning model in the embodiment of the present application, etc.) based on the training samples maintained in the database 530 to obtain the machine learning model / rule 501.

[0107] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0108] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the machine learning model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0109] The machine learning model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in Figure 2. The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc.

[0110] Specifically, the training device 520 may transfer the trained model to the execution device 510 .

[0111] In Figure 2, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (such as target data in the embodiment of the present application) into the I / O interface 512 through the client device 540.

[0112] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.

[0113] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0114] Finally, the I / O interface 512 provides the processed results to the client device 540 and thus to the user.

[0115] In the scenario shown in FIG. 2 , the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. Alternatively, client device 540 can automatically send input data to I / O interface 512. If user authorization is required for client device 540 to automatically send input data, the user can set the corresponding permissions in client device 540. The user can view the output of execution device 510 on client device 540, which can be presented in a display, sound, action, or other specific form. Client device 540 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, with I / O interface 512 directly storing the input data and output results from I / O interface 512 as new sample data in database 530.

[0116] It is worth noting that FIG2 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in FIG2 , the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 can be deployed in the client device 540.

[0117] From the inference side of the model:

[0118] In the embodiment of the present application, the computing module 511 of the above-mentioned execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in the embodiment of the present application.

[0119] In an embodiment of the present application, the computing module 511 of the execution device 510 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0120] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model reasoning process provided in the embodiment of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model reasoning process provided in the embodiment of the present application.

[0121] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model reasoning process provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 510, which is not limited here.

[0122] From the training side of the model:

[0123] In an embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 2, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training in the embodiment of the present application.

[0124] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0125] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.

[0126] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0127] (1) Neural Network

[0128] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0129] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0130] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way of extracting features is independent of position. The convolution kernel can be formalized as a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0131] CNN is a very common neural network. Convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the image input.

[0132] (3) Deep Neural Networks

[0133] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0134] (4) Loss function

[0135] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0136] (5) Backpropagation algorithm

[0137] The back propagation (BP) algorithm can be used to correct the size of the initial model parameters during training, reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial model, thereby converging the error loss. The BP algorithm is a backward propagation movement driven by error loss, aiming to obtain optimal model parameters, such as the weight matrix.

[0138] (6) Transformer Network

[0139] The transformer network may include an embedding layer and at least one transformer layer, wherein the at least one transformer layer may be N transformer layers (N is an integer greater than 0), wherein each transformer layer includes an attention layer, an add&norm layer, a feedforward layer, and an add&norm layer that are adjacent in sequence. In the embedding layer, the current input is embedded to obtain multiple feature vectors; in the attention layer, P input vectors are obtained from the previous layer of the transformer layer, and with any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained, thereby determining the P intermediate vectors corresponding to the P input vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in at least one transformer layer are used as feature representations of the current input.

[0140] Next, the above steps are introduced in detail with reference to specific examples.

[0141] First, in the embedding layer, the current input is embedded to obtain multiple feature vectors.

[0142] The embedding layer can be called an input embedding layer. The current input can be a text input, for example, a paragraph of text or a sentence. The text can be Chinese text, English text, or text in other languages. After obtaining the current input, the embedding layer can embed each word in the current input to obtain a feature vector for each word. In some embodiments, the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input to obtain a word embedding vector for each word. In the position encoding layer, the position of each word in the current input can be obtained, and then the position vector of each word can be processed. In some examples, the position of each word can be the absolute position of each word in the current input. Taking the current input "How many numbers should I return Huabei" as an example, the position of "how many" can be represented as the first position, the position of "number" can be represented as the second position, and so on. In some examples, the position of each word can be the relative position between each word. Still taking the current input of "On which date should I repay" as an example, the position of "on which date" can be expressed as before "number", the position of "number" can be expressed as after "on which date" and before "should", etc. When the word embedding vector and position vector of each word in the current input are obtained, the position vector of each word and the corresponding word embedding vector can be combined to obtain the feature vector of each word, that is, to obtain multiple feature vectors corresponding to the current input. Multiple feature vectors can be represented as an embedding matrix with a preset dimension. The number of feature vectors in the multiple feature vectors can be set to M, and the preset dimension can be H, so that the multiple feature vectors can be represented as an M×H embedding matrix.

[0143] (7) Attention mechanism

[0144] The attention mechanism mimics the internal process of biological observation behavior, namely, a mechanism that aligns internal experience and external sensations to increase the observation precision of certain areas. It can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features from sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of ​​the attention mechanism can be rewritten as the following formula:

[0145] Here, Lx = ||Source|| represents the length of the Source. This formula implies that the elements in the Source are imagined to consist of a series of data pairs. Given a Query element in the target, the similarity or correlation between the Query and each Key is calculated to obtain the weight coefficient for each Key's corresponding Value. The weighted sum of the Values ​​is then taken to obtain the final Attention value. Essentially, the Attention mechanism performs a weighted sum of the Values ​​of the Source elements, with the Query and Key used to calculate the weight coefficient for the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring the majority of less important information. This focusing process is reflected in the calculation of the weight coefficients: the larger the weight, the more focus is placed on the corresponding Value. In other words, the weight represents the importance of the information, while the Value represents the corresponding information. The self-attention mechanism can be understood as internal attention. The attention mechanism occurs between the Query element of the Target and all elements of the Source. The self-attention mechanism refers to the attention mechanism that occurs between the internal elements of the Source or the internal elements of the Target. It can also be understood as the attention calculation mechanism in the special case of Target = Source. The specific calculation process is the same, only the calculation object has changed.

[0146] The Transformer architecture, through its multi-head attention mechanism, can effectively simulate long-range dependencies. However, the computational complexity of the traditional Transformer architecture grows quadratically with the input size, making it difficult to directly apply to high-resolution image restoration tasks.

[0147] The computational complexity of the traditional Transformer architecture grows quadratically with the input size, making it difficult to directly apply to high-resolution image restoration tasks. Recent Transformer lightweighting methods, including the channel self-attention mechanism (CSA), compute self-attention along the channel dimension, achieving computational complexity that is linear with the resolution. However, CSA cannot establish direct dependencies between different spatial locations.

[0148] Therefore, it is urgent to design a network architecture that can capture the spatial dependencies between windows and establish direct dependencies between different spatial locations.

[0149] In order to solve the above problems, the present invention provides a data processing method. The data processing method of the present invention is described in detail below with reference to the accompanying drawings.

[0150] Refer to Figure 3, which is a flow chart of a data processing method provided in an embodiment of the present application. As shown in Figure 3, a data processing method provided in an embodiment of the present application may include steps 301 to 304, and these steps are described in detail below.

[0151] 301. Obtain target data;

[0152] The embodiments of the present application can be applied to the processing of a machine learning model, wherein the machine learning model may include a first network, a target network layer, and a second network.

[0153] The target network layer may be a transformer layer or other types of network layers. Specifically, the target network layer may be a network layer in a feature extraction network of a machine learning model.

[0154] The first network layer can be the network connected before the target network layer in the machine learning model, and the second network layer can be the network connected after the target network layer in the machine learning model. For example, the first network can include the input projection layer of the machine learning model and some network layers in the feature extraction network, and the second network can include some network layers in the feature extraction network of the machine learning model (when the target network layer is the last layer of the feature extraction network, the second network may not include the network layers in the feature extraction network of the machine learning model) and the task network including the output projection layer.

[0155] In one possible implementation, the target network can be the data input into the machine learning model.

[0156] For example, the target data is image data, and the machine learning model can be used to perform image processing tasks, and the processing result obtained by processing the target data through the machine learning model is the processing result of the image processing task.

[0157] For example, the target data is audio data, and the machine learning model can be used to perform an audio processing task, and the processing result obtained by processing the target data through the machine learning model is the processing result of the audio processing task.

[0158] For example, the target data is text data, and the machine learning model can be used to perform a text processing task, and the processing result obtained by processing the target data through the machine learning model is the processing result of the text processing task.

[0159] For example, image processing tasks may include image classification, target recognition, image enhancement (such as but not limited to super-resolution, denoising, rain removal, deblurring, demosaicing, etc.), image generation, and the like.

[0160] For example, audio processing tasks can include speech-to-text, audio enhancement, audio synthesis, and the like.

[0161] For example, text processing tasks can be summary generation, dialogue response, text generation and other tasks.

[0162] Next, the structure of the machine learning model in the embodiment of the present application will be introduced in conjunction with the accompanying drawings. As shown in Figure 4, an architectural diagram of the machine learning model is a U-shaped encoding-decoding network, comprising an input projection layer, a multi-layer stacked channel-spatial self-attention module (CSTB) and an output projection layer. Among them, the target network layer can have the structural features of the channel-spatial transformer module (CSTB). Figure 4 shows that the stacked transformer layers all have the structural features of the target network layer. It should be understood that one or more of the machine learning models in the embodiment of the present application can have the structural features of the target network layer, and the diagram of Figure 4 does not limit the present application.

[0163] Optionally, the input projection layer and the output projection layer can be a 3x3 convolution, and the channel-spatial Transformer module (CSTB) is composed of a context-based channel self-attention module (CB-CSA), a grid spatial self-attention module (G-SSA) in series, a re-parameterization locally-enhance feed-forward network (Rep-LeFFN) and a layer normalization module (layerNorm) in series.

[0164] In addition to the Unet network, the embodiments of the present application may also be applied to other network architectures, which is not limited by the present application.

[0165] 302. Obtain a first feature representation of the target data through the first network according to the target data;

[0166] 303. Based on the first feature representation, perform a feature extraction operation through the target network layer to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing an operation of a self-attention operation mechanism between different channels and performing an operation of a self-attention mechanism between different spaces.

[0167] The first feature representation may be a result obtained by a network connected before the target network layer, and the first feature representation may be used as an input of the target network layer.

[0168] In existing implementations, when implementing attention operations, the target network layer either calculates self-attention along the channel dimension, failing to establish a direct dependency between different spatial positions, or calculates self-attention within the spatial dimension, failing to clearly and completely capture the spatial dependencies between windows. In an embodiment of the present application, the self-attention operation mechanism between different channels and the self-attention mechanism between different spaces are simultaneously performed within the same network layer, combining the channel self-attention mechanism with the grid-based spatial self-attention mechanism, thereby establishing a direct dependency between different spatial positions, making up for the direct interaction between distant pixels that is missing in spatial self-attention, and capturing the spatial dependencies between windows.

[0169] Next, we introduce several structural diagrams of the target network layer:

[0170] In a possible implementation, the target network layer includes a first attention module and a second attention module. The first attention module can be used to perform a self-attention operation mechanism between different channels on the first feature representation; and the second attention module can be used to perform a self-attention operation mechanism between different spaces on the first feature representation after the self-attention operation mechanism between different channels.

[0171] That is to say, the first attention module in the target network layer for performing the self-attention operation mechanism between different channels can be connected in series with the second attention module for performing the self-attention operation mechanism between different spaces, and the first attention module is connected before the second attention module.

[0172] It should be understood that other modules may be connected between the first attention module and the second attention module.

[0173] For example, refer to Figure 5, which is a schematic diagram of the target network layer structure, wherein the first attention module (G-SSA) is before the second attention module (CB-CSA), and the first attention module and the second attention module may be connected with an FFN module, a layer regularization module, etc. The target network layer shown in Figure 5 can perform the following operations: X′ l=CB-CSA(LN(X l-1 ))+X l-1 , X l =Rep-LeFFN(LN(X′ l ))+X′ l ,

[0174] X′ l+1 =G-SSA(LN(X l ))+X l ,

[0175] X l+1 =Rep-LeFFN(LN(X′ l+1 ))+X′ l+1 ;

[0176] In a possible implementation, the target network layer includes a first attention module and a second attention module. The first attention module can be used to perform a self-attention mechanism operation between different spaces on the first feature representation; and the second attention module can be used to perform a self-attention operation mechanism operation between different channels on the first feature representation after the self-attention operation within the spatial dimension.

[0177] That is to say, the first attention module in the target network layer for performing the self-attention operation mechanism between different channels can be connected in series with the second attention module for performing the self-attention operation mechanism between different spaces, and the second attention module is connected before the first attention module.

[0178] It should be understood that other modules may be connected between the first attention module and the second attention module.

[0179] For example, referring to Figure 6, Figure 6 is a structural diagram of the target network layer, in which the first attention module (G-SSA) is after the second attention module (CB-CSA), and an FFN module, a layer regularization module, etc. can also be connected between the first attention module and the second attention module.

[0180] In one possible implementation, the target network layer includes a first processing branch and a second processing branch in parallel, the first processing branch includes a first attention module, the second processing branch includes a second attention module, and the processing result of the first processing branch is used to be fused with the processing result of the second processing branch.

[0181] That is to say, the first attention module in the target network layer used to perform the self-attention operation mechanism between different channels and the second attention module used to perform the self-attention operation mechanism between different spaces may not be connected in series, but belong to different branches in parallel.

[0182] For example, the first attention module and the second attention module can be connected in parallel. That is, the first attention module for performing the self-attention operation mechanism between different channels in the target network layer can be connected in parallel with the second attention module for performing the self-attention operation mechanism between different spaces. The first attention module can be used to perform the self-attention operation mechanism between different channels on the first feature representation; the second attention module can be used to perform the self-attention operation mechanism between different spaces on the first feature representation; and the first feature representation after the self-attention operation mechanism between different channels and the first feature representation after the self-attention operation within the spatial dimension are fused.

[0183] Exemplarily, referring to FIG7 , FIG7 is a schematic diagram of a structure of a target network layer, wherein the first attention module (G-SSA) and the second attention module (CB-CSA) are connected in parallel.

[0184] The channel-wise self-attention mechanism (CSA) computes self-attention along the channel dimension, achieving computational complexity that is linear with respect to resolution. The CSA is a spatially global self-attention computation method that can achieve a certain degree of global perspective. However, this global perspective is indirectly obtained by calculating the channel-wise self-attention matrix using a global scope. This results in a coarse global perspective that cannot accurately capture and establish global dependencies.

[0185] In an embodiment of the present application, in order to make the channel self-attention mechanism pay more attention to local information, the original global-based channel self-attention mechanism is improved to a local context-based channel self-attention mechanism.

[0186] Method 1:

[0187] In one possible implementation, when performing self-attention calculation in the channel dimension, the corresponding Q matrix, K matrix, and V matrix can be obtained according to the input features. For example, Query (Q), Key (K), and Value (V) matrices are generated respectively through three groups of 1x1 point convolutions followed by 3x3 depth-decomposable convolutions.

[0188] In one possible implementation, the Q matrix, K matrix, and V matrix of the input features can be divided separately to obtain multiple first Q matrices, multiple first K matrices, and multiple first V matrices; wherein the first Q matrix, the first K matrix, and the first V matrix correspond one-to-one, and the corresponding first Q matrix, the first K matrix, and the first V matrix have the same size. Unlike the original CSA, which directly calculates the attention map of Q and K and acts on V, the embodiment of the present application can divide the Q, K, and V matrices into windows of equal size along the spatial dimension.

[0189] In one possible implementation, multiple first calculation results can be obtained based on the multiple first Q matrices, the multiple first K matrices, and the multiple first V matrices, each of which is obtained through operation between a first Q matrix, a first K matrix, and a first V matrix with a corresponding relationship; the fusion result (for example, splicing) of the multiple first operation results is used to obtain the operation result of the self-attention operation mechanism between the different channels.

[0190] Method 2:

[0191] In one possible implementation, the Q matrix, K matrix, and V matrix of the input features can be shuffled and partitioned to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices. The second Q matrices, the second K matrices, and the second V matrices correspond one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size. Unlike Method 1, Method 1 is implemented by directly splitting the partitioned windows and calculating channels within non-overlapping windows. Method 2 first slides the window spatially and then calculates window-channel self-attention in the same manner. Similar to Method 1, Method 2 can also obtain multiple second calculation results based on the multiple second Q matrices, the multiple second K matrices, and the multiple second V matrices. Each second calculation result is obtained by operating on a corresponding second Q matrix, a second K matrix, and a second V matrix. The fusion result (e.g., concatenation) of the multiple second calculation results is used to obtain the result of the self-attention operation mechanism between the different channels.

[0192] In one possible implementation, the target network layer may include both the self-attention operation performed in the above-mentioned manner 1 and the self-attention operation performed in the above-mentioned manner 2. The self-attention operation performed in the above-mentioned manner 1 and the self-attention operation performed in the above-mentioned manner 2 may be performed respectively through two branches, and then the fusion result of the result obtained by the self-attention operation performed in the above-mentioned manner 1 (that is, the multiple first operation results in the above-mentioned embodiment) and the fusion result of the result obtained by the self-attention operation performed in the above-mentioned manner 2 (that is, the multiple second operation results in the above-mentioned embodiment) are fused.

[0193] In one possible implementation, the operation result of the self-attention operation mechanism between the different channels can be obtained by fusing the fusion result of multiple first operation results and the fusion result of the multiple second operation results (for example, weighted fusion).

[0194] Referring to Figure 8, Figure 8 shows an example architecture of CB-CSA. Similar to the original CSA, the feature map is first generated through three groups of 1x1 point convolutions followed by 3x3 depth-decomposable convolutions to generate the Query (Q), Key (K), and Value (V) matrices respectively. Next, unlike the original CSA that directly calculates the attention maps of Q and K and acts on V, the present invention divides the Q, K, and V matrices into windows of equal size along the spatial dimension. In order to enhance the contextual information, two parallel branches are constructed at the same time: one branch is directly based on the divided window split, and the channel attention map is calculated in non-overlapping windows and acts on V of the corresponding window; the other branch first slides the window in space, and then calculates the window channel self-attention in the same way. The outputs obtained by the upper and lower branches are multiplied by the corresponding masks and added together to obtain the final output.

[0195] The channel self-attention operation shown in Figure 8 can be expressed as the following operation: F out =Mask top ×F top +Mask bottom ×F bottom .

[0196] In existing implementations, dense spatial self-attention is combined with sparse spatial self-attention to establish global and local dependencies. However, the effectiveness and computational complexity of this method are still limited by the dense self-attention mechanism (computational complexity increases quadratically with the window), resulting in less than optimal overall results.

[0197] CB-CSA, due to the small windows in the space, leads to a severe islanding effect, where non-adjacent windows do not interact. However, the underlying visual task requires a global receptive field to achieve better results. To address this issue, the present embodiment introduces a grid spatial self-attention (G-SSA) module to provide information interaction between windows.

[0198] As shown in Figure 9A, Figure 9A shows a schematic diagram of a G-SSA module. Unlike the original SSA, G-SSA divides the global spatial dimension into equally spaced grids and calculates spatial self-attention between equally spaced grid points. 3) G-SSA decomposes the global SSA into pxp groups of global grid SSA along the spatial dimension, while reducing the computational effort by pxp times. The G-SSA module shown in Figure 9A can perform the following operations:

[0199] O(H 2 W 2 C)→O(H 2 W 2 C / p 2 )

[0200] Existing technologies use a local enhancement feedforward network to improve local feature extraction by inserting a 3x3 depthwise separable convolution between 1x1 convolutions. However, the receptive field of 3x3 depthwise separable convolution is limited, making it incapable of extracting larger-scale local features.

[0201] In an embodiment of the present application, multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations are connected after the nonlinear layer, so that local features of a larger scale can be extracted, which significantly enhances the performance of FFN.

[0202] In one possible implementation, the first network, the target network layer or the second network further includes: a feedforward network FFN; the FFN is used to perform multiple parallel depth-separable convolution operations and / or multiple serial point convolution operations on the result obtained by nonlinear activation; or, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation.

[0203] In one possible implementation, during model training, the FFN is used to perform multiple parallel depth-separable convolution operations and / or multiple serial point convolution operations on the results obtained by nonlinear activation; during model inference, the FFN is used to perform a target point convolution operation on the results obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

[0204] Referring to Figure 9B, Figure 9B is a schematic diagram of the structure of an FFN. Based on LeFFN, we introduce a reparameterization method to further enhance the local representation ability of LeFFN. As shown in Figure 9B, during the training phase, the representation ability of the network is enhanced by adding serial point convolutions and parallel depth-separable convolutions with various convolution kernel sizes. During the inference phase, the serial point convolution is reparameterized into a single point convolution, and the parallel multi-way depth-separable convolution is reparameterized into a depth-separable convolution with a large convolution kernel. As a result, the computational complexity of the inference phase is greatly reduced without losing the representation ability of the model.

[0205] The FFN shown in FIG9A can perform the following operations: FFN(x)=PW-Conv(GELU(PW-Conv(x))), LeFFN(x)=PW-Conv(DW-Conv(GELU(PW-Conv(x)))), RepPW-Conv(x)=PW-Conv((PW-Conv(x)), RepDW-Conv(x)=DW-Conv 5×5 (x)+DW-Conv 3×3 (x)+DW-Conv 1×1 (x)+x, Rep-LeFFN(x)=RepPW-Conv(RepDW-Conv(GELU(RepPW-Conv(x)))),

[0206] 304. Obtain a processing result of the target data through the second network according to the second feature representation.

[0207] Referring to Figure 9C, Figure 9C shows the FLOPS-PNSR curves of different models on Gaussian denoising, true denoising, and rain removal tasks. The results show that IPT-V2 surpasses existing SOTA solutions in multiple underlying visual tasks such as theoretical Gaussian denoising, true denoising, and rain removal, and achieves better FLOPS-PNSR curves. Referring to Table 1, Table 1 shows the ablation experiment of different modules on the Gaussian denoising task. As shown in Table 1, after replacing the original CSA with the CB-CSA of the present invention, the PSNR is significantly improved without a significant increase in Flops and Params. Table 2 shows the effects of the series, anti-series, and parallel structures of channel self-attention and grid self-attention. As shown in Table 2, compared with the Baseline (30.02dB), significant PSNR improvements have been achieved, verifying the effectiveness of the present invention.

[0208] Table 1

[0209] Table 2

[0210] 10 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG10 , a data processing device provided in an embodiment of the present application, the device 1000 includes:

[0211] Acquisition module 1001, used to acquire target data;

[0212] For a detailed description of the acquisition module 1001 , reference may be made to the description of step 301 in the above embodiment, which will not be repeated here.

[0213] The processing module 1002 is configured to obtain a first feature representation of the target data through the first network according to the target data; and perform a feature extraction operation through the target network layer according to the first feature representation to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing a self-attention operation mechanism between different channels and performing a self-attention operation mechanism between different spaces;

[0214] According to the second feature representation, a processing result of the target data is obtained through the second network.

[0215] For a detailed description of the processing module 1002 , reference may be made to the description of steps 302 to 304 in the above embodiment, which will not be repeated here.

[0216] In one possible implementation,

[0217] The target data is image data, and the processing result is the processing result of the image processing task; or,

[0218] The target data is audio data, and the processing result is the processing result of the audio processing task; or

[0219] The target data is text data, and the processing result is the processing result of the text processing task.

[0220] In a possible implementation, the target network layer includes a first attention module and a second attention module, and the processing module 1002 is specifically configured to:

[0221] performing, by the first attention module, a self-attention operation mechanism between different channels on the first feature representation;

[0222] Through the second attention module, the first feature representation after the self-attention operation mechanism between the different channels is operated by the self-attention mechanism between different spaces.

[0223] In a possible implementation, the target network layer includes a first attention module and a second attention module, and the processing module 1002 is specifically configured to:

[0224] Performing a self-attention mechanism operation between different spaces on the first feature representation through the first attention module;

[0225] Through the second attention module, the first feature representation after the self-attention operation in the spatial dimension is operated by the self-attention operation mechanism between different channels.

[0226] In a possible implementation, the processing module is specifically configured to:

[0227] performing a self-attention operation mechanism between different channels on the first feature representation through a first attention module;

[0228] Performing a self-attention mechanism operation between different spaces on the first feature representation through a second attention module;

[0229] Through the fusion module, the first feature representation after the operation of the self-attention operation mechanism between the different channels and the first feature representation after the self-attention operation within the spatial dimension are fused.

[0230] In one possible implementation, the operation of the self-attention operation mechanism between different channels includes:

[0231] Dividing the Q matrix, K matrix, and V matrix of the input features respectively to obtain a plurality of first Q matrices, a plurality of first K matrices, and a plurality of first V matrices; wherein the first Q matrices, the first K matrices, and the first V matrices correspond to each other one-to-one, and the corresponding first Q matrices, the first K matrices, and the first V matrices have the same size;

[0232] Based on the multiple first Q matrices, the multiple first K matrices and the multiple first V matrices, multiple first calculation results are obtained, each of the first calculation results is obtained through operation between the first Q matrix, the first K matrix and the first V matrix with a corresponding relationship; the fusion result of the multiple first operation results is used to obtain the operation result of the self-attention operation mechanism between the different channels.

[0233] In one possible implementation, the operation of the self-attention operation mechanism between different channels includes:

[0234] Scrambling and dividing the elements included in the Q matrix, K matrix, and V matrix of the input features respectively to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices; wherein the second Q matrices, the second K matrices, and the second V matrices correspond to each other one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size;

[0235] Obtaining a plurality of second calculation results based on the plurality of second Q matrices, the plurality of second K matrices, and the plurality of second V matrices, each of the second calculation results being obtained by operation between a second Q matrix, a second K matrix, and a second V matrix having a corresponding relationship;

[0236] The multiple first operation results are fused to obtain the operation result of the self-attention operation mechanism between the different channels.

[0237] In a possible implementation, the processing module 1002 is further configured to:

[0238] By fusing the fusion results of the multiple first operation results and the fusion results of the multiple second operation results, the operation results of the self-attention operation mechanism between the different channels are obtained.

[0239] In a possible implementation, the first network, the target network layer or the second network further includes: a feed-forward network FFN;

[0240] The FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; or,

[0241] The FFN is used to perform a target point convolution operation on the result obtained by the nonlinear activation.

[0242] In one possible implementation,

[0243] During model training, the FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation;

[0244] During model inference, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

[0245] Next, a terminal device provided in an embodiment of the present application is introduced. Please refer to Figure 11. Figure 11 is a structural diagram of a terminal device provided in an embodiment of the present application. The terminal device 1100 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Specifically, the terminal device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of processors 1103 in the terminal device 1100 can be one or more, and Figure 11 takes one processor as an example), wherein the processor 1103 may include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103 and the memory 1104 may be connected via a bus or other means.

[0246] The memory 1104 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1103. A portion of the memory 1104 may also include non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0247] Processor 1103 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are referred to as a bus system in the figure.

[0248] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1103. Processor 1103 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1103. The above processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1103 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1104. Processor 1103 reads information from memory 1104 and, in conjunction with its hardware, completes the steps involved in the model training or model inference process in the above method.

[0249] Receiver 1101 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1102 can be used to output digital or character information through the first interface. Transmitter 1102 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1102 can also include a display device such as a display screen.

[0250] The embodiment of the present application also provides a server. Please refer to Figure 12. Figure 12 is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1212 (for example, one or more processors) and a memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage medium 1230 can be temporary storage or permanent storage. The program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1212 can be configured to communicate with the storage medium 1230 to execute a series of instruction operations in the storage medium 1230 on the server 1200.

[0251] The server 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258; or one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0252] In an embodiment of the present application, the central processing unit 1212 is used to execute actions related to model training or model reasoning in the above embodiments.

[0253] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0254] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0255] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0256] Specifically, see Figure 13 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip may be a neural network processor (NPU) 1300. NPU 1300 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1303, which is controlled by controller 1304 to extract matrix data from memory and perform multiplication operations.

[0257] In some implementations, the arithmetic circuit 1303 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1303 is a two-dimensional systolic array. The arithmetic circuit 1303 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general-purpose matrix processor.

[0258] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 1302 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 1301 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 1308.

[0259] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 through the Direct Memory Access Controller (DMAC) 1305. Input data is also transferred to unified memory 1306 through the DMAC.

[0260] BIU stands for Bus Interface Unit 1310 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1309 .

[0261] The bus interface unit 1310 (BIU) is used for the instruction fetch memory 1309 to obtain instructions from the external memory, and is also used for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0262] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1306 or to move weight data to the weight memory 1302 or to move input data to the input memory 1301.

[0263] The vector calculation unit 1307 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0264] In some implementations, the vector calculation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function or a nonlinear function to the output of the operation circuit 1303, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1307 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1303, for example, for use in subsequent layers in a neural network.

[0265] An instruction fetch buffer 1309 connected to the controller 1304 is used to store instructions used by the controller 1304;

[0266] Unified memory 1306, input memory 1301, weight memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0267] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0268] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0269] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0270] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0271] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: Applied to processing of a machine learning model, the machine learning model including a first network, a target network layer, and a second network, the method comprising: Get target data; Obtaining, according to the target data, a first feature representation of the target data through the first network; performing a feature extraction operation on the target network layer based on the first feature representation to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing a self-attention operation between different channels and a self-attention operation between different spaces, and the target network layer is a network layer in the first network; According to the second feature representation, a processing result of the target data is obtained through the second network.

2. The method according to claim 1, characterized in that The target network layer is a transformer layer.

3. The method according to claim 1 or 2, characterized in that The target data is image data, and the processing result is the processing result of the image processing task; or, The target data is audio data, and the processing result is the processing result of the audio processing task; or The target data is text data, and the processing result is the processing result of the text processing task.

4. The method according to any one of claims 1 to 3, characterized in that: The target network layer includes a first attention module and a second attention module, and performing a feature extraction operation through the target network layer according to the first feature representation includes: performing, by the first attention module, a self-attention operation mechanism between different channels on the first feature representation; Through the second attention module, the first feature representation after the self-attention operation mechanism between the different channels is operated by the self-attention mechanism between different spaces.

5. The method according to any one of claims 1 to 3, characterized in that: The target network layer includes a first attention module and a second attention module, and performing a feature extraction operation through the target network layer according to the first feature representation includes: Performing a self-attention mechanism operation between different spaces on the first feature representation through the first attention module; Through the second attention module, the first feature representation after the self-attention operation in the spatial dimension is operated by the self-attention operation mechanism between different channels.

6. The method according to any one of claims 1 to 3, characterized in that: The target network layer includes a first attention module, a second attention module, and a fusion module. The feature extraction operation is performed through the target network layer according to the first feature representation, including: performing, by the first attention module, a self-attention operation mechanism between different channels on the first feature representation; Performing a self-attention mechanism operation between different spaces on the first feature representation through the second attention module; Through the fusion module, the first feature representation after the operation of the self-attention operation mechanism between the different channels and the first feature representation after the self-attention operation within the spatial dimension are fused.

7. The method according to any one of claims 1 to 6, characterized in that: The operation of the self-attention operation mechanism between different channels includes: Dividing the Q matrix, K matrix, and V matrix of the input features respectively to obtain a plurality of first Q matrices, a plurality of first K matrices, and a plurality of first V matrices; wherein the first Q matrices, the first K matrices, and the first V matrices correspond to each other one by one; Obtaining a plurality of first calculation results based on the plurality of first Q matrices, the plurality of first K matrices, and the plurality of first V matrices, each of the first calculation results being obtained by operation between a first Q matrix, a first K matrix, and a first V matrix having a corresponding relationship; The multiple first operation results are fused to obtain the operation result of the self-attention operation mechanism between the different channels.

8. The method according to any one of claims 1 to 6, characterized in that: The operation of the self-attention operation mechanism between different channels includes: Scrambling and dividing the elements included in the Q matrix, K matrix, and V matrix of the input features respectively to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices; wherein the second Q matrices, the second K matrices, and the second V matrices correspond to each other one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size; Based on the multiple second Q matrices, the multiple second K matrices and the multiple second V matrices, multiple second calculation results are obtained, each of the second calculation results is obtained through operation between the second Q matrix, the second K matrix and the second V matrix with a corresponding relationship; the fusion result of the multiple second operation results is used to obtain the operation result of the self-attention operation mechanism between the different channels.

9. The method according to claim 7 or 8, characterized in that The method further comprises: By fusing the fusion results of the multiple first operation results and the fusion results of the multiple second operation results, the operation results of the self-attention operation mechanism between the different channels are obtained.

10. The method according to any one of claims 1 to 9, characterized in that: The first network, the target network layer or the second network further includes: a feed-forward network FFN; The FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; or, The FFN is used to perform a target point convolution operation on the result obtained by the nonlinear activation.

11. The method according to claim 10, characterized in that During model training, the FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; During model inference, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

12. A data processing device, characterized in that: Applied to processing of a machine learning model, the machine learning model including a first network, a target network layer, and a second network, the apparatus comprising: Acquisition module, used to obtain target data; a processing module, configured to obtain, based on the target data, a first feature representation of the target data through the first network; and, based on the first feature representation, perform a feature extraction operation through the target network layer to obtain a second feature representation of the target data; wherein the feature extraction operation includes performing a self-attention operation mechanism between different channels and performing a self-attention operation mechanism between different spaces, and the target network layer is a network layer in the first network; According to the second feature representation, a processing result of the target data is obtained through the second network.

13. The device according to claim 12, characterized in that The target data is image data, and the processing result is the processing result of the image processing task; or, The target data is audio data, and the processing result is the processing result of the audio processing task; or The target data is text data, and the processing result is the processing result of the text processing task.

14. The device according to claim 12 or 13, characterized in that The target network layer includes a first attention module and a second attention module, and the processing module is specifically configured to: performing, by the first attention module, a self-attention operation mechanism between different channels on the first feature representation; Through the second attention module, the first feature representation after the self-attention operation mechanism between the different channels is operated by the self-attention mechanism between different spaces.

15. The device according to claim 12 or 13, characterized in that The target network layer includes a first attention module and a second attention module, and the processing module is specifically configured to: Performing a self-attention mechanism operation between different spaces on the first feature representation through the first attention module; Through the second attention module, the first feature representation after the self-attention operation in the spatial dimension is operated by the self-attention operation mechanism between different channels.

16. The device according to claim 12 or 13, characterized in that The processing module is specifically used to: performing a self-attention operation mechanism between different channels on the first feature representation through a first attention module; Performing a self-attention mechanism operation between different spaces on the first feature representation through a second attention module; Through the fusion module, the first feature representation after the operation of the self-attention operation mechanism between the different channels and the first feature representation after the self-attention operation within the spatial dimension are fused.

17. The device according to any one of claims 12 to 16, characterized in that The operation of the self-attention operation mechanism between different channels includes: Dividing the Q matrix, K matrix, and V matrix of the input features respectively to obtain a plurality of first Q matrices, a plurality of first K matrices, and a plurality of first V matrices; wherein the first Q matrices, the first K matrices, and the first V matrices correspond to each other one-to-one, and the corresponding first Q matrices, the first K matrices, and the first V matrices have the same size; Obtaining a plurality of first calculation results based on the plurality of first Q matrices, the plurality of first K matrices, and the plurality of first V matrices, each of the first calculation results being obtained by operation between a first Q matrix, a first K matrix, and a first V matrix having a corresponding relationship; The multiple first operation results are fused to obtain the operation result of the self-attention operation mechanism between the different channels.

18. The device according to any one of claims 12 to 16, characterized in that The operation of the self-attention operation mechanism between different channels includes: Scrambling and dividing the elements included in the Q matrix, K matrix, and V matrix of the input features respectively to obtain multiple second Q matrices, multiple second K matrices, and multiple second V matrices; wherein the second Q matrices, the second K matrices, and the second V matrices correspond to each other one-to-one, and the corresponding second Q matrices, second K matrices, and second V matrices have the same size; Based on the multiple second Q matrices, the multiple second K matrices and the multiple second V matrices, multiple second calculation results are obtained, each of the second calculation results is obtained through operation between the second Q matrix, the second K matrix and the second V matrix with a corresponding relationship; the fusion result of the multiple second operation results is used to obtain the operation result of the self-attention operation mechanism between the different channels.

19. The device according to claim 17 or 18, characterized in that The processing module is further configured to: By fusing the fusion results of the multiple first operation results and the fusion results of the multiple second operation results, the operation results of the self-attention operation mechanism between the different channels are obtained.

20. The device according to any one of claims 12 to 19, characterized in that The first network, the target network layer or the second network further includes: a feed-forward network FFN; The FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; or, The FFN is used to perform a target point convolution operation on the result obtained by the nonlinear activation.

21. The device according to claim 20, characterized in that During model training, the FFN is used to perform multiple parallel depth-wise separable convolution operations and / or multiple serial point-wise convolution operations on the results obtained by nonlinear activation; During model inference, the FFN is used to perform a target point convolution operation on the result obtained by nonlinear activation, and the target point convolution operation is equivalent to the multiple parallel depth-separable convolution operations and / or the multiple serial point convolution operations after training.

22. A computer storage medium, characterized in that The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 11.

23. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device, cause the computer device to execute the method according to any one of claims 1 to 11.

24. A system comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other; The at least one memory is used to store code; The at least one processor is configured to execute the code to perform the method according to any one of claims 1 to 11.

25. A chip comprising a processor, characterized in that: The processor is used to support a data processing device to implement the method according to any one of claims 1 to 11.

Images