Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method, and the method comprises the steps: processing to-be-processed data through an attention network of an encoder when the to-be-processed data is processed through any encoder of a neural network, and obtaining a first correlation matrix obtained by the corresponding attention network according to the to-be-processed data; and determining n target data components from the m data components according to the first correlation matrix obtained by each attention network. Thus, the target data component with the attention information meeting the preset requirement (for example, the attention value is relatively high) can be determined from the to-be-processed data, so that the attention information of the target data component is dynamically adjusted to obtain the output of the corresponding encoder, for example, the target data component with the relatively high attention value can be weakened or even shielded, and the output of the encoder is improved. Therefore, abnormal information is prevented from being transmitted into a subsequent processing layer, the influence of an adversarial sample is weakened, and the robustness of the neural network is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and related equipment. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] In actual application scenarios, neural networks may be subject to adversarial attacks, that is, slight perturbations to the input can cause abnormal behavior of the neural network. For example, it can cause a neural network performing a classification task to misclassify, or cause a neural network performing a text translation task to translate a sentence with a completely different meaning.

[0004] Currently, one method to improve the robustness of neural networks is to perform adversarial training on them. Adversarial training usually requires repeatedly calling neural networks to generate adversarial samples for specific scenarios. This process consumes a lot of resources, and the generated adversarial samples and corresponding adversarial training are limited by the usage scenarios, which means that the robustness of neural networks can only be improved in specific usage scenarios. Summary of the invention

[0005] The embodiment of the present application provides a data processing method, which can conveniently and efficiently improve the robustness of neural networks in various application scenarios. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0006] In a first aspect, the present application provides a data processing method, which includes: obtaining data to be processed, wherein the data to be processed includes m data components, where m is an integer greater than 1; processing the data to be processed by at least one encoder in a neural network to obtain a data processing result, wherein each encoder includes at least one attention network; when the data to be processed is processed by any encoder of at least one encoder, the method includes: processing the data to be processed by the attention network of the encoder to obtain a first correlation matrix obtained by the corresponding attention network according to the data to be processed, the first correlation matrix is ​​used to describe the correlation between the query matrix and the key matrix of the data to be processed in the corresponding attention network; determining n target data components from the m data components according to the first correlation matrix obtained by each attention network, where n is a positive integer and n is not greater than m; obtaining the attention result of each attention network according to the n target data components and the first correlation matrix obtained by each attention network; obtaining the output of the encoder according to the attention result of each attention network in the encoder, and the output of the encoder is used to obtain the data processing result.

[0007] In the first aspect, a dynamic attention mechanism is proposed, in which the target data component whose attention information meets the preset requirements (for example, higher attention) can be determined from the data to be processed, so as to dynamically adjust the attention information of the target data component. For example, the target data component with a higher attention value can be weakened or even masked, so as to avoid abnormal information from being transmitted to subsequent processing layers, thereby reducing the impact of adversarial samples and improving the robustness of the neural network.

[0008] In a possible implementation of the first aspect, n target data components are determined from m data components according to a first correlation matrix obtained by each attention network, including: determining an attention value corresponding to each data component according to the first correlation matrix obtained by each attention network; determining n target data components from the m data components according to the attention value corresponding to each data component.

[0009] In this possible implementation, there may be multiple ways to determine n target data components from m data components.

[0010] In one example, in a text classification task, the attention values ​​of m data components may be sorted, and then the first n data components with the highest attention values ​​may be selected as target data components.

[0011] In another example, in a text generation task, n data components after the first a data components with the highest attention values ​​are selected as target data components, that is, data components ranked from a-th to b-th in attention values ​​can be selected as target data components, where ba=n-1, and a is greater than 1 and b is not greater than m.

[0012] In a possible implementation of the first aspect, the attention result of each attention network is obtained based on n target data components and a first correlation matrix obtained by each attention network, including: in the first correlation matrix obtained by any attention network, adjusting the elements corresponding to the n target data components to obtain an adjusted first correlation matrix; and obtaining the attention result of the corresponding attention network based on the adjusted first correlation matrix.

[0013] In this possible implementation, illustratively, one column in the first correlation matrix corresponds to one data component, and therefore, in the first correlation matrix, an element corresponding to a target data component is a column corresponding to the target data component in the first correlation matrix. In some scenarios, the elements corresponding to the target data components in the first correlation matrix can be reduced to weaken or even mask the attention information of this part of the data component. The adjustment amplitudes of the elements corresponding to different target data components can be the same or different. In addition, the elements corresponding to the same target data component can be adjusted partially or completely.

[0014] In a possible implementation of the first aspect, in a first correlation matrix obtained by any attention network, elements corresponding to n target data components are adjusted to obtain an adjusted first correlation matrix, including: in a first correlation matrix obtained by any attention network, elements corresponding to n target data components are adjusted according to corresponding preset coefficients to obtain an adjusted first correlation matrix, and the preset coefficients are less than 1.

[0015] In this possible implementation, the value of the preset coefficient may be in various situations. For example, in a text classification task, β may be 0, and in a text generation task, β may be 0.5.

[0016] In this way, the data components with higher attention values ​​can be weakened or even masked to prevent abnormal information from being passed to the next layer, thereby reducing the impact of adversarial samples.

[0017] In a possible implementation of the first aspect, the output of the encoder is obtained according to the attention results of each attention network in the encoder, including: performing a first processing according to the attention results of each attention network in the encoder to obtain a first tensor; setting a first proportion of elements in the first tensor to a first preset value to obtain a second tensor; and obtaining the output of the encoder according to the second tensor.

[0018] In this possible implementation, a random dropout mechanism can be used to further improve the subsequent processing of the attention results in the encoder to further enhance the robustness of the neural network.

[0019] In a possible implementation manner of the first aspect, the elements of the first proportion in the second tensor are randomly selected.

[0020] In a possible implementation of the first aspect, obtaining the output of the encoder according to the second tensor includes: performing a second processing on the second tensor to obtain a third tensor; setting a second proportion of elements in the third tensor to a second preset value to obtain a fourth tensor; and obtaining the output of the encoder according to the fourth tensor.

[0021] In this possible implementation, in the process of obtaining the output of the encoder according to the second tensor, a random deactivation mechanism may be used for further improvement.

[0022] In a possible implementation manner of the first aspect, the elements of the second proportion in the third tensor are randomly selected.

[0023] The second aspect of the present application provides a data processing device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module and a processing module.

[0024] The third aspect of the present application provides an electronic device, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0025] The fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0026] The fifth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0027] In a sixth aspect of the present application, a chip system is provided, which includes a processor for supporting an electronic device to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the electronic device. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0028] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided by an embodiment of the present application;

[0030] Figure 2a This is a schematic diagram of an application scenario of natural language processing provided by an embodiment of the present application;

[0031] Figure 2b is a schematic diagram of another application scenario of natural language processing provided by an embodiment of the present application;

[0032] Figure 2c This is another schematic diagram of an application scenario of natural language processing provided by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of an exemplary structure of a neural network provided in an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;

[0035] Figure 5 It is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;

[0036] Figure 6 is a schematic diagram of an exemplary structure of a neural network provided in an embodiment of the present application;

[0037] Figure 7 is an exemplary schematic diagram of the adjusted first correlation matrix provided in an embodiment of the present application;

[0038] Figure 8 is a schematic diagram of an embodiment of a data processing device provided in an embodiment of the present application;

[0039] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0041] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0042] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0043] First, the overall workflow of the artificial intelligence system is described.

[0044] See also Figure 1 , Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0045] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0046] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0047] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0048] (1) Infrastructure

[0049] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (central processing unit (CPU), graphics processing unit (GPU), neural-network processing unit (NPU), tensor processing unit (TPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA) and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0050] (2) Data

[0051] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0052] (3) Data processing

[0053] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0054] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0055] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0056] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0057] (4) General ability

[0058] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0059] (5) Smart products and industry applications

[0060] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0061] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0062] (1) Neural Network

[0063] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0064]

[0065] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0066] (2) Loss Function

[0067] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0068] (3) Back propagation algorithm

[0069] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0070] (4) Attention Network

[0071] Attention network is a neural network that uses attention mechanism. In deep learning, attention mechanism can be broadly defined as a weight vector that describes importance: this weight vector is used to predict or infer an element. For example, for a pixel in an image or a word in a sentence, the correlation between the target element and other elements can be quantitatively estimated using the attention vector, and the weighted sum of the attention vectors is used as the approximation of the target.

[0072] The attention mechanism in deep learning simulates the attention mechanism of the human brain. For example, when humans look at a painting, although the human eye can see the whole picture of the whole painting, when humans observe it in depth and carefully, the eyes actually focus on only a part of the pattern in the whole painting. At this time, the human brain mainly focuses on this small pattern. In other words, when humans observe an image carefully, the human brain does not pay equal attention to the whole image, but has a certain weight distinction. This is the core idea of ​​the attention mechanism.

[0073] Simply put, the human visual processing system tends to selectively focus on certain parts of the image and ignore other irrelevant information, which helps the human brain perceive. Similarly, in the attention mechanism of deep learning, in some problems involving language, speech or vision, some parts of the input may be more relevant than other parts. Therefore, through the attention mechanism in the attention network, the attention network can dynamically focus on the part of the input that helps to effectively perform the current task.

[0074] (5) Self-Attention Network

[0075] The self-attention network is a neural network that uses the self-attention mechanism. The self-attention mechanism is an extension of the attention mechanism. The self-attention mechanism is actually an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. The self-attention mechanism can play a key role in machine reading, abstract summarization, or image description generation.

[0076] Taking the application of self-attention network in natural language processing as an example, the self-attention network processes input data of arbitrary length and generates a new feature expression of the input data, and then converts the feature expression into the target word. The self-attention network layer in the self-attention network uses the attention mechanism to obtain the relationship between all other words, thereby generating a new feature expression for each word. The advantage of the self-attention network is that the attention mechanism can directly capture the relationship between all words in a sentence without considering the position of the words.

[0077] In actual application scenarios, neural networks may be subject to adversarial attacks, that is, slight perturbations to the input can cause abnormal behavior of the neural network. For example, it can cause a neural network performing a classification task to misclassify, or cause a neural network performing a text translation task to translate a sentence with a completely different meaning.

[0078] Currently, one method to improve the robustness of neural networks is to perform adversarial training on the neural networks.

[0079] However, adversarial training relies on the fragility of neural networks to generate adversarial samples. For neural networks, especially large-scale neural networks, the process of generating adversarial samples requires repeated calls to the neural network, which consumes a lot of resources. In addition, in order to learn the knowledge in the adversarial samples, it is usually necessary to update the parameters of the neural network according to the adversarial samples, which may reduce the generalization ability of the neural network. In addition, adversarial training depends on specific downstream tasks. If the neural network is a pre-trained large-scale neural network, and the pre-trained large-scale neural network is usually obtained by unsupervised training, therefore, adversarial training will be limited by the usage scenario, that is, it can only improve the robustness of the neural network in a specific usage scenario.

[0080] Another way to improve the robustness of neural networks is to perform defense distillation. For example, the basic model can be trained with training data and corresponding soft labels, and then the distillation model can be trained with the training data and the probability distribution output by the trained basic model. However, defense distillation requires the training of two neural networks and relies on the soft labels given by downstream tasks. It can be seen that defense distillation is also limited by the usage scenario, so that the obtained trained distillation model is limited to a specific usage scenario. In addition, defense distillation can usually only defend against specific attack algorithms, resulting in a poor degree of defense.

[0081] It can be seen that the current methods of improving the robustness of neural networks are highly dependent on the quality of the training data itself, and the training process is relatively complicated. In addition, they are also limited by the usage scenarios, which means that they can only improve the robustness of neural networks in specific usage scenarios.

[0082] Based on the above problems, an embodiment of the present application provides a data processing method that can conveniently and efficiently improve the robustness of neural networks in a variety of application scenarios.

[0083] The data processing method can be applied to a variety of application scenarios.

[0084] For example, the data processing method can be applied to a natural language processing system.

[0085] In one example, the data processing method can be applied to a natural language processing scenario, where a neural network is used to perform natural language processing, and the corresponding data to be processed includes text.

[0086] In another example, the data processing method can be applied to an image processing scenario, in which case a neural network is used to perform image processing, and the corresponding data to be processed includes images. For example, the data to be processed can be data in the form of images, image sequences, or videos.

[0087] There may be many electronic devices that execute the data processing method.

[0088] The following uses a natural language processing scenario as an example to exemplify an electronic device that executes the data processing method.

[0089] In the natural language processing scenario, this data processing method can be applied to the natural language processing system.

[0090] like Figure 2a As shown, a natural language processing system includes a user device and a data processing device.

[0091] User devices include users and smart terminals such as mobile phones, personal computers or information processing centers. User devices are the initiators of natural language data processing. As the initiator of language questions or queries, users usually initiate requests through user devices.

[0092] Data processing devices can be devices or servers with data processing functions such as cloud servers, network servers, application servers, and management servers. Data processing devices receive query statements / voice / text and other questions from smart terminals through interactive interfaces, and then perform language data processing in the form of machine learning, deep learning, search, reasoning, and decision-making through the memory that stores data and the processor link for data processing. Memory can be a general term, including local storage and databases that store historical data. The database can be on the data processing device or on other network servers.

[0093] In this example, the electronic device executing the data processing method may be Figure 2a Data processing equipment in.

[0094] like Figure 2b The figure shows another application scenario of the natural language processing system. In this scenario, the smart terminal directly acts as a data processing device, directly receiving input from the user and directly processing it by the hardware of the smart terminal itself. The specific process is similar to Figure 2a Similarly, please refer to the above description and will not repeat them here.

[0095] In this example, the electronic device executing the data processing method may be Figure 2b Smart terminals in.

[0096] like Figure 2c As shown, the user device may be a terminal device 201 or a terminal device 202, and the data processing device may be a server 210, wherein the data storage system 250 may be integrated on the server 210, or may be set on a cloud or other network server.

[0097] The structure of the neural network in the embodiment of the present application is introduced below.

[0098] like Figure 3As shown, the neural network in the embodiment of the present application may include but is not limited to at least P encoders, where P is a positive integer, that is, the neural network includes at least one encoder.

[0099] When a neural network includes multiple encoders, there can be multiple connection relationships between the multiple encoders, which are not limited here.

[0100] For example, in one example, the multiple encoders may be multiple cascaded encoders, and in another example, the multiple encoders may also include at least two parallel encoders.

[0101] The structures of the multiple encoders may be the same or different. In the embodiment of the present application, one of the encoders is taken as an example to exemplify the structure and related processing steps of the encoder, and any other encoder in the neural network may be the same or similar to the encoder.

[0102] Specifically, if Figure 3 As shown, any encoder in the neural network may include at least one attention network.

[0103] Among them, the attention network is implemented based on the attention mechanism. The specific structure of the attention network is introduced in the subsequent method embodiments in combination with the specific processing steps, and will not be repeated here.

[0104] The number and structure of the attention network in the encoder can be various and are not limited here.

[0105] For example, in one example, any encoder in a neural network may include multiple parallel attention networks.

[0106] In addition, in some examples, the neural network may also include other structures besides the at least one encoder, which is not limited here. Exemplarily, the neural network may also include one or more of an embedding layer, a decoder, a classifier, etc.

[0107] In the embodiments of the present application, there may be many specific application scenarios of the neural network, which are not limited here.

[0108] For example, the neural network can be used for text processing, in which case the input of the neural network includes text. Exemplarily, the neural network can be a network improved based on the BERT network, etc., and the text processing tasks performed by the neural network can be one or more of the following tasks: machine translation, automatic summary generation, opinion extraction, text classification, and question answering.

[0109] Alternatively, the neural network can be used for image processing, in which case the input of the neural network includes an image, for example, an image, an image sequence, or a video, etc. Exemplarily, the neural network can be improved based on networks such as ViT and VisualBERT, and the image processing tasks performed by the neural network can include one or more of the following tasks: image classification, target detection, image enhancement, and image segmentation.

[0110] The following is a detailed introduction to the specific structure of the attention network in the neural network and the corresponding data processing steps in conjunction with the data processing method embodiment.

[0111] Among them, the data processing method embodiment can be executed in the reasoning stage corresponding to the neural network, or it can be executed in the training stage corresponding to the neural network, and the specific application scenario of the embodiment of the present application is not limited here.

[0112] For ease of description, the data processing method embodiment is exemplarily introduced below by taking the natural language processing task performed in the reasoning stage as an example.

[0113] like Figure 4 As shown, the data processing method embodiment includes steps 401-402.

[0114] Step 401, obtaining data to be processed.

[0115] The data to be processed includes m data components, where m is an integer greater than 1.

[0116] There may be multiple sources of the data to be processed. For example, the data to be processed may be locally stored in the electronic device executing the embodiment of the present application, or may be transmitted to the electronic device by other devices, or may be collected by the electronic device through a web page or other applications, or may be obtained after the electronic device processes image data, voice data, etc. There may also be multiple specific types of the data to be processed, which are not limited here. In some examples, the data processing method is applied to the scenario of natural language processing, and the data to be processed includes text. In another example, the data processing method can be applied to the scenario of image processing, and the data to be processed includes images. For example, the data to be processed may be data in the form of images, image sequences, or videos.

[0117] In the reasoning scenario corresponding to the adversarial attack, the data to be processed can be considered as adversarial data in the adversarial attack.

[0118] In the embodiment of the present application, m data components can be determined from the data to be processed.

[0119] For example, if the data to be processed is text data, the data to be processed can be segmented using existing or subsequently developed word segmentation technology to obtain m data components in the data to be processed.

[0120] For example, in one example, the data to be processed is “G reat For Winter Time: Good alternative to those really cold winter days. When you want something grilled fast. I wish it had more fixtures, plus its bulky.” The original sample corresponding to the data to be processed is “Great For Winter Time: Good alternative to those really cold winter days. When you want something grilled fast. I wish it had more attachments, plus its bulky.” The correct text type corresponding to the original sample is positive emotion.

[0121] After obtaining the data to be processed, the data to be processed can be segmented to obtain a sequence after segmentation ['G', 're', '##at', 'For', 'Winter', 'Time', ':', 'Good', 'alternative', ...]. The beginning and end of the sequence will be marked with two tags [CLS] and [SEP] respectively to indicate the text classification and mark the end position of the text.

[0122] In another example, the data to be processed is "I don't want to stay at this hotel anymore. The bathroom is dirty and hard to find. It's really annoying." The original sample corresponding to the data to be processed is "I don't want to stay at this hotel anymore. The bathroom is dirty and hard to find. It's really annoying." The correct category is negative comments.

[0123] After obtaining the data to be processed, the data to be processed can be segmented to obtain a sequence after segmentation ['not', 'willing', 'again', 'live', 'this', 'home', ',', 'health', 'health', 'room', ...]. Two indication marks [CLS] and [SEP] are added to the beginning and end of the sequence respectively to indicate text classification and mark the end position of the text.

[0124] The data component in the embodiment of the present application is the data component other than the special mark in the sequence corresponding to the data to be processed. For example, the data component does not include the indicator marks [CLS] and [SEP], the mask [MASK], and the prefix mark in the prompt learning, etc.

[0125] If the data to be processed includes an image, the image may be segmented using a preset image segmentation method to obtain m data components.

[0126] Step 402: Process the data to be processed by at least one encoder in the neural network to obtain a data processing result.

[0127] A data processing result can be obtained according to the processing result of at least one encoder in the neural network on the data to be processed.

[0128] The neural network may include but is not limited to the at least one encoder, and may also process the data to be processed through other layers.

[0129] For example, a neural network may include an embedding layer.

[0130] Specifically, through the embedding layer, the word embedding vector (word embedding) and positional embedding (PE) corresponding to the data to be processed can be obtained. By adding the corresponding word embedding vector and positional encoding, the representation vector corresponding to the data to be processed can be obtained as the input of the subsequent layer. Exemplarily, the word embedding vector is obtained by pre-training using algorithms such as Word2Vec and Glove. The positional encoding can be obtained using a function in the form of a trigonometric function. In this way, the representation vector corresponding to the data to be processed not only realizes the vector representation of the text, but also saves information about the relative or absolute position of the word in the sequence.

[0131] After obtaining the representation vector corresponding to the data to be processed through the embedding layer, the representation vector can be input into at least one encoder to be processed by the at least one encoder to obtain a corresponding output.

[0132] For example, the neural network may include multiple cascaded encoders, and the representation vector corresponding to the data to be processed can be input into the first encoder of the multiple cascaded encoders, and then processed in sequence through the multiple cascaded encoders to obtain the output of the last encoder, so as to obtain the corresponding data processing result based on the output of the last encoder.

[0133] In different scenarios, the specific content of the data processing result may be different. For example, in the natural language processing scenario, the data processing result is a text processing result, for example, when performing a text classification task, the data processing result indicates the text category of the data to be processed; and in the image processing scenario, the data processing result is an image processing result, for example, when performing an object detection task, the data processing result indicates the object category in the data to be processed.

[0134] like Figure 5 As shown, specifically, when the data to be processed is processed by any encoder of at least one encoder, the method includes steps 501-504.

[0135] Step 501, processing the data to be processed through the attention network of the encoder to obtain a first correlation matrix obtained by the corresponding attention network according to the data to be processed.

[0136] The first correlation matrix is ​​used to describe the correlation between the query matrix and the key matrix of the data to be processed in the corresponding attention network.

[0137] The attention network in the encoder can calculate the query matrix (query, Q), key matrix (key, K) and value matrix (value, V) corresponding to the data to be processed in the attention network according to the corresponding input X. The specific calculation formula can be:

[0138] Q=XW Q

[0139] K=XW K

[0140] V=XW V

[0141] Among them, W Q is the query weight matrix, W K is the key weight matrix, W V is the value weight matrix, W Q , W K and W V It can be determined through training.

[0142] When the neural network includes multiple cascaded encoders, for the first encoder, X in the encoder is the representation vector corresponding to the data to be processed; for encoders other than the first encoder, X in the encoder is the output of the previous encoder.

[0143] In this way, the corresponding first correlation matrix can be calculated according to the query matrix Q and the key matrix K corresponding to the data to be processed in the attention network.

[0144] For example, for attention network k, the calculation formula of the first correlation matrix A_k obtained by the attention network k is as follows:

[0145]

[0146] Among them, Q k is the query matrix of attention network k, K k is the key matrix of the attention network k, and d is the dimension of the query matrix and the key matrix.

[0147] When there are multiple parallel attention networks in the encoder, the first correlation matrix corresponding to each attention network can be obtained.

[0148] Step 502, determining n target data components from the m data components according to the first correlation matrix obtained by the corresponding attention network.

[0149] n is a positive integer, and n is not greater than m.

[0150] In the embodiment of the present application, the first correlation matrix obtained by the attention network can reflect the correlation information between the features. Therefore, based on the correlation information between the features, n key target data components whose correlation meets the requirements can be determined from the m data components. In different scenarios, the determination method of the n key target data components can be different.

[0151] In some embodiments, step 502 specifically includes the following steps:

[0152] Determine an attention value corresponding to each data component according to a first correlation matrix obtained by each attention network;

[0153] According to the attention value corresponding to each data component, n target data components are determined from the m data components.

[0154] Among them, there are multiple ways to determine the attention value corresponding to each data component based on the first correlation matrix obtained by each attention network.

[0155] For example, in some examples, the encoder includes multiple parallel attention networks. Then, the first correlation matrices obtained by each attention network can be added together to obtain a global matrix A. The global matrix A can be considered as the global correlation matrix in the encoder. The specific calculation formula can be:

[0156] A=∑A k

[0157] For example, each value in the global matrix A reflects the total attention of the query tag of the data component to the key tag of the data component.

[0158] In this way, the attention value corresponding to each data component can be determined according to the global matrix A.

[0159] For example, the total attention value or average attention value corresponding to a certain data component can be calculated based on the global matrix A as the attention value of the data component.

[0160] For example, each column in the global matrix A can be summed, and the specific calculation formula is as follows:

[0161] A s =∑ i A[i,j]=[x1,x2,……xm]

[0162] Wherein, m is the number of data components, and xi is the total attention value of the i-th data component. In this example, the total attention value corresponding to the i-th data component is used as the attention value of the i-th data component. In other examples, the average attention value xi / m corresponding to the i-th data component can also be calculated as the attention value of the i-th data component.

[0163] In this way, the attention value corresponding to each data component can be determined.

[0164] Then, n target data components can be determined from the m data components according to the attention value corresponding to each data component.

[0165] There are many ways to determine the number and selection method of n.

[0166] In one example, the values ​​of n corresponding to different encoders in the neural network may be the same or different.

[0167] For example, in different encoders, n can be randomly selected from any integer in [2,4].

[0168] Furthermore, there may be multiple ways to determine n target data components from the m data components.

[0169] In one example, in a text classification task, the attention values ​​of m data components may be sorted, and then the first n data components with the highest attention values ​​may be selected as target data components.

[0170] In another example, in a text generation task, n data components after the first a data components with the highest attention values ​​are selected as target data components, that is, data components ranked from a-th to b-th in attention values ​​can be selected as target data components, where ba=n-1, and a is greater than 1 and b is not greater than m.

[0171] Step 503, obtaining the attention result of each attention network according to the n target data components and the first correlation matrix obtained by each attention network.

[0172] Compared with conventional attention results, in an embodiment of the present application, the attention results of each attention network can be dynamically adjusted based on n target data components, so that the attention information of the target data component can be dynamically adjusted, thereby reducing the impact of adversarial samples.

[0173] Among them, after determining the n target data components, the first correlation matrix obtained by the attention network can be adjusted according to the n target data components to obtain the attention result of the attention network.

[0174] In some embodiments, obtaining an attention result of each attention network according to the n target data components and the first correlation matrix obtained by each attention network includes:

[0175] In a first correlation matrix obtained by any attention network, adjusting elements corresponding to n target data components to obtain an adjusted first correlation matrix;

[0176] According to the adjusted first correlation matrix, the attention result of the corresponding attention network is obtained.

[0177] For example, in the scenario of text processing tasks, a column in the first correlation matrix corresponds to a data component. Therefore, in the first correlation matrix, an element corresponding to a target data component is a column corresponding to the target data component in the first correlation matrix.

[0178] In the embodiment of the present application, there may be multiple ways to adjust the elements corresponding to the n target data components in the first correlation matrix.

[0179] In some scenarios, the elements corresponding to the target data components in the first correlation matrix can be reduced to weaken or even mask the attention information of this part of the data component; or, in other scenarios, the elements corresponding to the target data components can also be increased.

[0180] The adjustment ranges of the elements corresponding to different target data components may be the same or different. In addition, the elements corresponding to the same target data component may be adjusted partially or completely.

[0181] In some embodiments, in a first correlation matrix obtained by any attention network, adjusting elements corresponding to n target data components to obtain an adjusted first correlation matrix includes:

[0182] In a first correlation matrix obtained by any attention network, the elements corresponding to the n target data components are adjusted according to corresponding preset coefficients to obtain an adjusted first correlation matrix, where the preset coefficients are less than 1.

[0183] Specifically, according to the first correlation matrix A k [i,j], obtain the adjusted first correlation matrix A′ k [i,j] can be described by the following formula:

[0184]

[0185] in, is a set of n target data components, and β is a preset coefficient.

[0186] In the embodiment of the present application, the value of the preset coefficient can be in various situations. Among them, β can be set to [0, 1], where β = 0 means that the attention value is completely blocked, and β = 1 means that the attention value is not modified.

[0187] For example, in a text classification task, β can be 0, while in a text generation task, β can be 0.5.

[0188] In this way, the data components with higher attention values ​​can be weakened or even masked to prevent abnormal information from being passed to the next layer, thereby reducing the impact of adversarial samples.

[0189] After obtaining the adjusted first correlation matrix, the attention result of the corresponding attention network can be obtained according to the adjusted first correlation matrix.

[0190] For each attention network, the adjusted first correlation matrix A′ obtained by the attention network can be k And the value matrix V calculated in the attention network k Multiply them together to obtain the attention result of the attention network.

[0191] Step 504, obtaining the output of the encoder according to the attention results of each attention network in the encoder.

[0192] The output of the encoder is used to obtain the data processing result.

[0193] Among them, according to the attention results of each attention network in the encoder, the specific steps of obtaining the output of the encoder can be determined by the specific structure of the encoder.

[0194] For example, in one example, the encoder includes multiple parallel attention networks, and the attention results obtained by each attention network can be concatenated and multiplied by the corresponding weight matrix (also called projection) to obtain the intermediate representation H in the encoder. The intermediate representation H can be calculated by the following formula:

[0195] H=Concat k (A′ k ·V k )·W

[0196] Then, the output of the encoder can be obtained based on the intermediate representation H.

[0197] Exemplarily, the encoder may include a residual layer, a normalization layer, and a feedforward network, so that the intermediate representation H may be sequentially passed through the residual layer, the normalization layer, the feedforward network, etc. to obtain the output of the encoder.

[0198] It can be seen that in an embodiment of the present application, a dynamic attention mechanism is proposed, in which the target data component whose attention information meets the preset requirements (for example, higher attention) can be determined from the data to be processed, so that the attention information of the target data component can be dynamically adjusted. For example, the target data component with a higher attention value can be weakened or even masked, so as to avoid abnormal information from being transmitted to subsequent processing layers, thereby reducing the impact of adversarial samples and improving the robustness of the neural network.

[0199] In addition, in some embodiments, a random deactivation mechanism can be used to further improve the subsequent processing of the attention results in the encoder to further enhance the robustness of the neural network.

[0200] Specifically, in some embodiments, according to the attention results of each attention network in the encoder, obtaining the output of the encoder according to the attention results, obtaining the output of the encoder includes:

[0201] Perform a first processing on the attention result according to the attention result of each attention network in the encoder to obtain a first tensor;

[0202] Set the first proportion of elements in the first tensor to a first preset value to obtain a second tensor;

[0203] Based on the second tensor, get the output of the encoder.

[0204] Exemplarily, the first processing may include stitching, projection, residual processing and / or normalization processing.

[0205] For example, the attention results obtained by each attention network can be concatenated and multiplied by the corresponding weight matrix (also called projection) to obtain the intermediate representation H in the encoder, which can be calculated by the following formula:

[0206] H=Concat k (A′ k ·V k )·W

[0207] Then, residual processing and normalization can be performed on the intermediate representation H to obtain the first tensor H′. The specific formula is as follows:

[0208] H′=LayerNorm(H+z)

[0209] Among them, z is the input of the corresponding attention network.

[0210] Then, the first proportion of elements in the first tensor may be set to a first preset value to obtain a second tensor.

[0211] The specific value of the first ratio is not limited here. For example, the first ratio may be 5%, 10%, etc.

[0212] In some examples, the first proportion of elements in the second tensor are selected randomly.

[0213] There may be many specific values ​​of the first preset value. Generally speaking, the first preset value may be between [0, 1]. In one example, the first preset value may be 0.

[0214] The second tensor The calculation formula can be shown as follows:

[0215]

[0216] Among them, Bernoulli (0.1) is a Bernoulli random variable, specifically a random variable that satisfies mutual independence and takes only two values ​​of 0 and 1.

[0217] In this way, according to the second tensor To obtain the output of the encoder, for example, the second tensor can be sequentially input into a feed-forward network including a fully connected layer, a residual layer, and a layer normalization layer, etc.

[0218] In addition, in some examples, in the process of obtaining the output of the encoder according to the second tensor, a random deactivation mechanism can be used for further improvement.

[0219] Specifically, in some embodiments, obtaining the output of the encoder according to the second tensor includes:

[0220] Perform a second process on the second tensor to obtain a third tensor;

[0221] Set the elements of the second scale in the third tensor to the second preset value to obtain a fourth tensor;

[0222] Based on the fourth tensor, get the output of the encoder.

[0223] Exemplarily, the second processing may include processing through a feed-forward network, a residual layer, and / or a normalization layer.

[0224] The specific value of the second ratio is not limited here either, and by way of example, the second ratio may be 5%, 10%, etc. The second ratio may be the same as the first ratio or may be different.

[0225] In some examples, a second proportion of elements in the third tensor are selected randomly.

[0226] There are many specific cases for the second preset value. Generally speaking, the second preset value may be between [0, 1]. In one example, the second preset value may be 0. The second preset value may be the same as or different from the first preset value.

[0227] In some examples, according to the second tensor The process of obtaining the output of the encoder can be described by the following formula:

[0228]

[0229]

[0230]

[0231] in, is the second tensor, FFN(·) refers to the fully connected layer, which can be used as a feedforward network, O refers to the output of the fully connected layer, LayerNorm(·) refers to the layer normalization process, O′ is the third tensor, Bernoulli(0.1) is the Bernoulli random variable, is the fourth tensor.

[0232] In this way, the fourth tensor can be used as the output of the encoder, and the fourth tensor can be further processed to obtain the output of the encoder.

[0233] When a neural network includes multiple cascaded encoders, each encoder can be improved by using the above-mentioned dynamic attention mechanism and random inactivation mechanism to enhance the robustness of the neural network.

[0234] like Figure 6 The figure shows an exemplary structural diagram of the neural network.

[0235] The neural network includes P encoders, where P is a positive integer.

[0236] In each encoder, there are multiple parallel attention networks. For the convenience of description, Figure 6 The multi-head attention network in includes these multiple parallel attention networks.

[0237] The multiple parallel attention networks can adopt the dynamic attention mechanism in any of the above embodiments.

[0238] After obtaining the output of the multi-head attention network, the output of the multi-head attention network can be input into the residual layer and the normalization layer, and residual operation and normalization processing are performed, and then the random deactivation operation in any of the above embodiments is performed.

[0239] Next, it can be processed in sequence through a feedforward network, a residual layer, and a normalization layer, and then the random deactivation operation in any of the above embodiments can be performed to obtain the output of the corresponding encoder.

[0240] An exemplary specific application of an embodiment of the present application is described below through a specific example.

[0241] In one example, the neural network includes multiple cascaded encoders, each encoder including 12 parallel attention networks.

[0242] The following introduces a processing process for the data to be processed, respectively, based on the dynamic attention mechanism and the random inactivation mechanism.

[0243] 1. Dynamic Attention Mechanism

[0244] In this example, the neural network can be improved on the basis of the bert-large-cased model.

[0245] The data to be processed as adversarial data is “G reat For Winter Time: Good alternative to those really cold winter days. When you want something grilled fast. I wish it had more fixtures, plus its bulky.” denoted as x. The original input data corresponding to the data to be processed is “Great For Winter Time: Good alternative to those really cold winter days. When you want something grilled fast. I wish it had more attachments, plus its bulky.” The correct text category is positive sentiment.

[0246] In addition, the preset coefficient β=0 can be preset, and the value range of the target data component n can be determined to be an integer between 3 and 6.

[0247] After obtaining the data to be processed x, the data to be processed can be segmented to obtain a sequence after segmentation ['G', 're', '##at', 'For', 'Winter', 'Time', ':', 'Good', 'alternative', ...]. Two special tags [CLS] and [SEP] will be added to the beginning and end of the sequence respectively. The two special tags are used to indicate text classification and mark the end position of the text.

[0248] Then, the 39 data components in the sequence can be word embedded to obtain the corresponding word vector, and the word vector can be input into the first encoder.

[0249] The 12 attention networks in the encoder are transformed by radiation, based on their respective W Q , W K and W V , calculate the query matrix Q, key matrix K and value matrix V corresponding to the data to be processed in the corresponding attention network.

[0250] According to the query matrix Q and key matrix K in each attention network, the 39*39 first correlation matrix of each attention network can be calculated.

[0251] By adding the 12 first correlation matrices, a global matrix can be obtained. Then, each column in the global matrix can be summed. The sum of a column is the attention value of a corresponding data component. In this way, the attention values ​​of the 39 data components can be obtained.

[0252] In the encoder, the value of n can be randomly selected as 4. The 39 attention values ​​are sorted, and the 4 data components with the highest attention values ​​are used as target data components.

[0253] In this example, the four target data components may be 'Time', 'alternative', 'cold' and 'fixtures'.

[0254] The preset coefficient β=0, then the columns corresponding to the four target data components in each first correlation matrix are set to 0.

[0255] like Figure 7 An exemplary schematic diagram of the adjusted first correlation matrix is ​​shown.

[0256] Among them, you can see that the attention values ​​corresponding to 'Time', 'alternative', 'cold' and 'fixtures' are set to 0.

[0257] Then, the 12 adjusted first correlation matrices may be multiplied with their corresponding value matrices respectively, and then the multiplication results may be concatenated and other operations may be performed to obtain an intermediate representation H, which includes 39 vectors with a length of 1024.

[0258] 2. Random Dropout Mechanism

[0259] After obtaining the intermediate representation H, a residual operation can be performed on the intermediate representation H, that is, the intermediate representation H is added to the input of the attention network, and then normalized and randomly deactivated.

[0260] In the random deactivation operation, 10% of the elements (i.e., 102) in the obtained 1024-dimensional vector can be randomly selected and their values ​​are set to 0. This operation is performed once for all 39 vectors concatenated in the intermediate representation. In this way, the second tensor can be obtained.

[0261] The second tensor passes through the fully connected layer and undergoes residual operation, layer normalization operation and random dropout operation again.

[0262] Similarly, in the random deactivation process, 10% of the elements (ie, 39*102) of the 39 1024-dimensional vectors are randomly selected and their values ​​are set to 0. In this way, the output of the first-layer encoder can be obtained.

[0263] The output of this encoder can be input into the next layer of encoder for processing.

[0264] The processing process of subsequent encoders of other layers is similar to the processing process of the encoder of the first layer. For details, please refer to the processing process of the encoder of the first layer, which will not be repeated here.

[0265] It should be noted that in different encoders of the neural network, the values ​​of the preset coefficient β and the target data component n may be the same or different, the number and specific objects of the target data components determined in different encoders may be different, and the adjustment method of the corresponding first correlation matrix based on the preset coefficient β may also be different. For example, in the last layer of encoders, the number of target data components may be 2, and the determined target data components may include data components different from the target data components 'Time', 'alternative', 'cold' and 'fixtures' determined in the first encoder.

[0266] After obtaining the output of the last encoder, the output of the last encoder may be input into a classifier (eg, a fully connected layer) to obtain a data processing result, which indicates that the text category of the data to be processed is positive emotion.

[0267] In the above embodiment, improvements to the neural network when the neural network is used to perform related tasks in the inference stage are introduced. That is, the above dynamic attention mechanism and random inactivation mechanism can be used to improve the processing process of the neural network in the inference stage.

[0268] During the training phase of the neural network, the above-mentioned dynamic attention mechanism and / or random dropout mechanism may or may not be used.

[0269] For example, in some examples, the dynamic attention mechanism is only used in the inference stage. Considering that the pre-training of existing large-scale neural networks usually consumes a lot of resources, the dynamic attention mechanism is not used in the pre-training stage. In the inference stage, the dynamic attention mechanism is used to optimize the trained large-scale neural network to improve the robustness of the existing trained large-scale neural network.

[0270] Furthermore, in some examples, the dropout mechanism can be applied both in the training phase and in the inference phase.

[0271] It can be seen that the above embodiments at least achieve the following technical effects:

[0272] 1. It can dynamically identify data components whose attention information meets preset requirements (e.g., high attention value), thus avoiding transmission of abnormal information to subsequent processing layers.

[0273] 2. The data components whose attention information meets the preset requirements (such as high attention values) can be weakened or even masked, thereby reducing the impact of adversarial samples.

[0274] The purpose of reducing the attention values ​​corresponding to data components whose attention meets preset requirements (for example, higher attention values) in the dynamic attention mechanism is to reduce the attention of all data components in the data to be processed on these target data components, thereby reducing the erroneous influence of data components whose attention values ​​meet the preset requirements on the remaining data components.

[0275] In addition, a residual operation can be added to the encoder. For example, after executing the dynamic attention mechanism to obtain the attention result, a residual operation can be added in the first processing and / or the second processing to retain the information transmitted by the upper layer through the residual operation, so that the information of the data component that is weakened or even obscured by the attention can be retained. Therefore, the dynamic attention mechanism has little effect on the accuracy of the neural network. In this way, the robustness of the neural network can be improved while ensuring the processing accuracy of the neural network.

[0276] 3. Through the random deactivation mechanism, the impact of adversarial attacks is weakened. In addition, the use of dynamic attention mechanism combined with random deactivation mechanism can greatly reduce the impact of adversarial attacks.

[0277] 4. Compared with the prior art, the embodiments of the present application do not require the type of attack and do not need to use the prior knowledge of adversarial samples, and greatly reduce the success rate of attacks, increase the number of calls required for attacking neural networks, and improve the classification accuracy of adversarial samples. In addition, the power consumption of training and reasoning is improved very little, and it is easy to deploy. In addition, the embodiments of the present application are applicable to various attack algorithms without adding additional training overhead.

[0278] The data processing method provided in the embodiments of the present application is described above from multiple aspects. The data processing device provided in the embodiments of the present application is described below in conjunction with the accompanying drawings.

[0279] like Figure 8 As shown, the embodiment of the present application provides a data processing device 80, and the data processing device 80 includes:

[0280] An acquisition module 801 is used to acquire data to be processed, where the data to be processed includes m data components, where m is an integer greater than 1;

[0281] A processing module 802 is used to process the data to be processed by at least one encoder in the neural network to obtain a data processing result, wherein each encoder includes at least one attention network;

[0282] When the data to be processed is processed by any encoder of at least one encoder, the processing module 802 is used to:

[0283] Processing the data to be processed by the attention network of the encoder to obtain a first correlation matrix obtained by the corresponding attention network according to the data to be processed, wherein the first correlation matrix is ​​used to describe the correlation between the query matrix and the key matrix of the data to be processed in the corresponding attention network;

[0284] According to the first correlation matrix obtained by each attention network, n target data components are determined from the m data components, where n is a positive integer and n is not greater than m;

[0285] Obtaining an attention result of each attention network according to the n target data components and a first correlation matrix obtained by each attention network;

[0286] According to the attention results of each attention network in the encoder, the output of the encoder is obtained, and the output of the encoder is used to obtain the data processing result.

[0287] Optionally, the processing module 802 is used to:

[0288] Determine an attention value corresponding to each data component according to a first correlation matrix obtained by each attention network;

[0289] According to the attention value corresponding to each data component, n target data components are determined from the m data components.

[0290] Optionally, the processing module 802 is used to:

[0291] In a first correlation matrix obtained by any attention network, adjusting elements corresponding to n target data components to obtain an adjusted first correlation matrix;

[0292] According to the adjusted first correlation matrix, the attention result of the corresponding attention network is obtained.

[0293] Optionally, the processing module 802 is used to:

[0294] In a first correlation matrix obtained by any attention network, the elements corresponding to the n target data components are adjusted according to corresponding preset coefficients to obtain an adjusted first correlation matrix, where the preset coefficients are less than 1.

[0295] Optionally, the processing module 802 is used to:

[0296] Perform a first processing according to the attention result of each attention network in the encoder to obtain a first tensor;

[0297] Set a first proportion of elements in the first tensor to a first preset value to obtain a second tensor;

[0298] Based on the second tensor, get the output of the encoder.

[0299] Optionally, the first scale of elements in the second tensor are selected randomly.

[0300] Optionally, the processing module 802 is used to:

[0301] Perform a second process on the second tensor to obtain a third tensor;

[0302] Set the elements of the second scale in the third tensor to the second preset value to obtain a fourth tensor;

[0303] Based on the fourth tensor, get the output of the encoder.

[0304] Optionally, the second scale of elements in the third tensor are selected randomly.

[0305] Fig. 9 , which is a possible logical structure diagram of an electronic device 90 provided in an embodiment of the present application. The electronic device 90 is used to implement the functions of the electronic device involved in any of the above embodiments. The electronic device 90 includes: a memory 901, a processor 902, a communication interface 903 and a bus 904. Among them, the memory 901, the processor 902, and the communication interface 903 are connected to each other through the bus 904.

[0306] The memory 901 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 901 may store a program. When the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute one or more steps of steps 301-303 of the above-mentioned data processing method embodiment.

[0307] The processor 902 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing related programs to implement the functions required to be executed by the acquisition module and the processing module in the data processing device in the above embodiment, or to execute one or more steps in the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 901, and the processor 902 reads the information in the memory 901, and executes one or more steps in the above data processing method embodiment in combination with its hardware.

[0308] The communication interface 903 uses a transceiver device such as but not limited to a transceiver to implement communication between the electronic device 90 and other devices or a communication network.

[0309] The bus 904 can realize a path for transmitting information between various components of the electronic device 90 (for example, the memory 901, the processor 902, and the communication interface 903). The bus 904 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0310] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above Fig. 9 The steps performed by the processor in .

[0311] In another embodiment of the present application, a computer program product is further provided, the computer program product comprising computer execution instructions, the computer execution instructions being stored in a computer readable storage medium; when the processor of the device executes the computer execution instructions, the device executes the above Fig. 9 The steps performed by the processor in .

[0312] In another embodiment of the present application, a chip system is further provided, the chip system comprising a processor, the processor being used to implement the above Fig. 9 In one possible design, the chip system may also include a memory, which is used to store the necessary program instructions and data for the device to write data. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0313] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0314] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0315] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0316] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0317] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire data to be processed, wherein the data to be processed includes m data components, where m is an integer greater than 1; Processing the data to be processed by at least one encoder in a neural network to obtain a data processing result, wherein each encoder includes at least one attention network; When the data to be processed is processed by any encoder of the at least one encoder, the method includes: Processing the data to be processed by the attention network of the encoder to obtain a first correlation matrix obtained by the corresponding attention network according to the data to be processed, wherein the first correlation matrix is ​​used to describe the correlation between the query matrix and the key matrix of the data to be processed in the corresponding attention network; Determine n target data components from the m data components according to the first correlation matrix obtained by each attention network, where n is a positive integer and n is not greater than m; Obtaining an attention result of each attention network according to the n target data components and a first correlation matrix obtained by each attention network; According to the attention result of each attention network in the encoder, the output of the encoder is obtained, and the output of the encoder is used to obtain the data processing result.

2. The method according to claim 1, characterized in that: The step of determining n target data components from the m data components according to the first correlation matrix obtained by each attention network comprises: Determining an attention value corresponding to each of the data components according to a first correlation matrix obtained by each attention network; According to the attention value corresponding to each of the data components, n target data components are determined from the m data components.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining the attention result of each attention network according to the n target data components and the first correlation matrix obtained by each attention network includes: In a first correlation matrix obtained by any attention network, adjusting the elements corresponding to the n target data components to obtain an adjusted first correlation matrix; According to the adjusted first correlation matrix, an attention result of the corresponding attention network is obtained.

4. The method according to claim 3, characterized in that The step of adjusting the elements corresponding to the n target data components in the first correlation matrix obtained by any attention network to obtain an adjusted first correlation matrix includes: In a first correlation matrix obtained by any attention network, the elements corresponding to the n target data components are adjusted according to corresponding preset coefficients to obtain an adjusted first correlation matrix, where the preset coefficients are less than 1.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining the output of the encoder according to the attention result of each attention network in the encoder comprises: Performing a first processing according to the attention result of each attention network in the encoder to obtain a first tensor; Setting a first proportion of elements in the first tensor to a first preset value to obtain a second tensor; An output of the encoder is obtained according to the second tensor.

6. The method according to claim 5, characterized in that The first proportion of elements in the second tensor are selected randomly.

7. The method according to claim 5 or 6, characterized in that: The step of obtaining the output of the encoder according to the second tensor includes: Performing a second process on the second tensor to obtain a third tensor; Setting a second proportion of elements in the third tensor to a second preset value to obtain a fourth tensor; An output of the encoder is obtained according to the fourth tensor.

8. The method according to claim 7, characterized in that A second proportion of elements in the third tensor are selected randomly.

9. A data processing device, characterized in that: include: An acquisition module, used for acquiring data to be processed, wherein the data to be processed includes m data components, where m is an integer greater than 1; A processing module, configured to process the data to be processed by at least one encoder in a neural network to obtain a data processing result, wherein each encoder includes at least one attention network; When the data to be processed is processed by any encoder of the at least one encoder, the processing module is used to: Processing the data to be processed by the attention network of the encoder to obtain a first correlation matrix obtained by the corresponding attention network according to the data to be processed, wherein the first correlation matrix is ​​used to describe the correlation between the query matrix and the key matrix of the data to be processed in the corresponding attention network; Determine n target data components from the m data components according to the first correlation matrix obtained by each attention network, where n is a positive integer and n is not greater than m; Obtaining an attention result of each attention network according to the n target data components and a first correlation matrix obtained by each attention network; According to the attention result of each attention network in the encoder, the output of the encoder is obtained, and the output of the encoder is used to obtain the data processing result.

10. The device according to claim 9, characterized in that The processing module is used for: Determining an attention value corresponding to each of the data components according to a first correlation matrix obtained by each attention network; According to the attention value corresponding to each of the data components, n target data components are determined from the m data components.

11. The device according to claim 9 or 10, characterized in that The processing module is used for: In a first correlation matrix obtained by any attention network, adjusting the elements corresponding to the n target data components to obtain an adjusted first correlation matrix; According to the adjusted first correlation matrix, an attention result of the corresponding attention network is obtained.

12. The device according to claim 11, characterized in that The processing module is used for: In a first correlation matrix obtained by any attention network, the elements corresponding to the n target data components are adjusted according to corresponding preset coefficients to obtain an adjusted first correlation matrix, where the preset coefficients are less than 1.

13. The device according to any one of claims 9 to 12, characterized in that: The processing module is used for: Performing a first processing according to the attention result of each attention network in the encoder to obtain a first tensor; Setting a first proportion of elements in the first tensor to a first preset value to obtain a second tensor; An output of the encoder is obtained according to the second tensor.

14. The device according to claim 13, characterized in that The first proportion of elements in the second tensor are selected randomly.

15. The device according to claim 13 or 14, characterized in that The processing module is used for: Performing a second process on the second tensor to obtain a third tensor; Setting a second proportion of elements in the third tensor to a second preset value to obtain a fourth tensor; An output of the encoder is obtained according to the fourth tensor.

16. The device according to claim 15, characterized in that A second proportion of elements in the third tensor are selected randomly.

17. An electronic device, characterized in that: The electronic device includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 1-8.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images