Sequence processing method and device, computing equipment and program product

Abstract

The invention discloses a sequence processing method and device, computing equipment and a program product, and belongs to the technical field of AI. According to the method and the device, the self-attention among the elements in each segment can be calculated for the plurality of segments included in the input sequence, so that the in-segment self-attention containing the local context information is obtained. In addition, feature information of a plurality of segments can be extracted, and self-attention between each element and the plurality of segments is calculated by using the feature information of the plurality of segments, so that inter-segment self-attention containing global context information is obtained. On this basis, a self-attention output result obtained based on the intra-segment self-attention and the inter-segment self-attention of the plurality of elements simultaneously contains local context information of each element in the segment and global context information of each element. Therefore, the precision of the self-attention output result is improved on the premise of reducing the calculation amount and the memory overhead of the self-attention calculation as much as possible.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a sequence processing method, apparatus, computing device, and program product. Background Technology

[0002] Currently, most AI models, such as large language models (LLM), computer vision (CV) models, and multimodal models, are based on the transformer architecture. The transformer architecture, through its self-attention mechanism, enables efficient modeling of the relationship between any two elements in the input sequence. This allows the AI ​​model to fully consider the context of the input sequence when performing prediction tasks, thereby improving the accuracy of the prediction task.

[0003] In related technologies, to reduce the computational cost and memory overhead of self-attention calculations, the input sequence can be divided into multiple segments, and self-attention between elements within each segment can be calculated to obtain the self-attention output corresponding to the input sequence. However, in this self-attention calculation method, because the input sequence is divided into multiple segments and self-attention calculations are performed at the segment level, the self-attention output only contains local contextual information of elements within each segment. Therefore, the accuracy of the self-attention output is not high, resulting in low accuracy of the predicted output sequence based on this self-attention output. Summary of the Invention

[0004] This application provides a sequence processing method, apparatus, computing device, and program product, which can improve the accuracy of self-attention output results while minimizing the computational load and memory overhead of self-attention computation, thereby improving the prediction accuracy of the output sequence.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] A first aspect provides a sequence processing method, the method comprising: dividing an input sequence into multiple segments, the input sequence including multiple elements, each segment including at least two elements, the multiple elements in the input sequence being used to represent multiple data units in the data to be processed, the multiple elements corresponding one-to-one with the multiple data units; calculating self-attention between elements within each segment to obtain intra-segment self-attention corresponding to each element; extracting feature information of the multiple segments, and calculating self-attention between each element in the input sequence and the multiple segments based on the feature information of the multiple segments to obtain inter-segment self-attention corresponding to each element in the input sequence; and determining a self-attention output result based on the intra-segment self-attention and inter-segment self-attention corresponding to the multiple elements in the input sequence.

[0007] In this input sequence, multiple elements have a sequential order, and each segment includes at least two consecutive elements. Furthermore, each element in the input sequence represents a data unit in the data to be processed.

[0008] In this application, for an input sequence comprising multiple segments, self-attention between elements within each segment can be calculated, resulting in intra-segment self-attention containing local context information. Furthermore, feature information from multiple segments can be extracted and used to calculate self-attention between each element and multiple segments, resulting in inter-segment self-attention containing global context information. Based on this, the self-attention output obtained from intra-segment and inter-segment self-attention for multiple elements will simultaneously contain both the local context information of each element within the segment and the global context information of each element. Therefore, this application improves the accuracy of the self-attention output while minimizing the computational and memory overhead of self-attention calculation, thereby improving the accuracy of the output sequence predicted based on this self-attention output.

[0009] Optionally, in this application, the input sequence is used to represent the sequence of data to be processed, and each element in the input sequence is used to represent a data unit in the data to be processed. The multiple data units in the data to be processed have a sequential order; for example, the multiple data units may have a chronological order.

[0010] For example, the data to be processed is text, the data unit is a word in the text, and the multiple elements are feature vectors of multiple words in the text; or, the data to be processed is an image, the data unit is an image region in the image, and the multiple elements are feature vectors of multiple image regions in the image; or, the data to be processed is video, the data unit is a video unit in the video, and the multiple elements are feature vectors of multiple video units in the video; or, the data to be processed is speech data, the data unit is a speech segment in the speech data, and the multiple elements are feature vectors of multiple speech segments in the speech data. Therefore, the sequence processing method provided in this application can be applied to various scenarios such as text processing, image processing, video processing, and speech processing, to reduce the computational load and memory overhead of AI models calculating self-attention in these scenarios.

[0011] Optionally, the feature information of each segment includes a feature key vector and a feature value vector; the process of extracting the feature information of the multiple segments may include: performing feature extraction on the key matrix of the first segment among the multiple segments to obtain the feature key vector of the first segment, wherein the key matrix of the first segment includes the key vectors of the elements in the first segment; performing feature extraction on the value matrix of the first segment to obtain the feature value vector of the first segment, wherein the value matrix of the first segment includes the value vectors of the elements in the first segment.

[0012] In this application, feature extraction is performed on the key matrix of the fragment, and the resulting feature key vector can be used to represent the core content of the fragment's key matrix. Similarly, feature extraction is performed on the value matrix of the fragment, and the resulting eigenvalue vector can be used to represent the core content of the fragment's value matrix. Representing the fragment using feature key vectors and eigenvalue vectors can reduce subsequent computational and memory overhead.

[0013] Optionally, the process of calculating the self-attention between each element in the input sequence and the multiple segments based on the feature information of the multiple segments to obtain the inter-segment self-attention corresponding to each element in the input sequence may include: performing self-attention calculation based on the query matrix of the input sequence, the feature key vectors and feature value vectors of the multiple segments to obtain the inter-segment self-attention corresponding to each element in the input sequence, wherein the query matrix of the input sequence includes the query vectors of the elements in the input sequence.

[0014] In this application, self-attention calculation is performed using the query matrix of the input sequence and the feature key vectors and feature value vectors of multiple segments. This can capture the association between each element in the input sequence and multiple segments. In this way, not only can the global context information of each element be obtained, but also, since it is the association between each element and the segment that is calculated, the amount of computation is small, thus achieving a balance between accuracy and computation.

[0015] Optionally, the process of calculating the self-attention between elements within each segment to obtain the segment self-attention corresponding to each element may include: performing linear self-attention calculation based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments to obtain the segment self-attention corresponding to each element in the first segment, wherein the query matrix of the first segment includes the query vector of the element in the first segment, the key matrix of the first segment includes the key vector of the element in the first segment, and the value matrix of the first segment includes the value vector of at least two elements in the first segment.

[0016] The query matrix and key matrix of the first segment can refer to the original query matrix and original key matrix obtained by projecting the feature matrix of the first segment. Alternatively, the query matrix and key matrix of the first segment can include the query matrix and key matrix obtained by positionally encoding the original query matrix and original key matrix based on the positions of the elements in the first segment in the input sequence.

[0017] In this application, linear self-attention computation can be performed based on the query matrix, key matrix, and value matrix of a fragment to obtain the intra-segment self-attention for each element within the fragment. Compared to traditional self-attention computation, linear self-attention computation has lower complexity, lower computational cost, and lower memory overhead. Therefore, using a linear self-attention mechanism to compute the intra-segment self-attention for each element can further reduce computational cost and memory overhead.

[0018] Optionally, the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix. The method further includes: based on the position of each element in the first segment in the input sequence, using a first positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively, to obtain the first query matrix and the first key matrix; based on the position of each element in the first segment in the input sequence, using a second positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively, to obtain the second query matrix and the second key matrix.

[0019] In this application, when calculating self-attention between elements within each segment, a dual positional encoding operator (i.e., a first positional encoding operator and a second positional encoding operator) is used to positionally encode the query matrix and key matrix of each segment based on the position of each element within the input sequence. Each vector in the encoded query matrix and key matrix will contain the absolute position information of the corresponding element. Based on this, when calculating self-attention using the encoded query matrix and key matrix, the correlation between elements can be captured more accurately; that is, a more precise correlation or similarity between elements can be obtained, thereby improving the accuracy of the self-attention calculation results.

[0020] Optionally, the implementation process of performing linear self-attention calculation based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments may include: obtaining the positive and negative matrices corresponding to the query matrix of the first segment and the positive and negative matrices corresponding to the key matrix of the first segment, wherein the positive matrix includes positive elements in the corresponding matrix and the negative matrix includes negative elements in the corresponding matrix; determining the absolute value matrix of the negative matrix corresponding to the query matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the key matrix of the first segment, wherein the absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix; and performing linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix.

[0021] In this application, the positive and negative elements in the query matrix and key matrix can be split into two matrices to obtain the corresponding positive and negative matrices. The absolute values ​​of the elements in the negative matrix containing negative elements are then taken. Linear self-attention calculation is then performed using the positive matrix containing positive elements and the absolute value matrix containing negative elements. In this way, the information represented by the negative elements in the query matrix and key matrix can be preserved, thereby ensuring the accuracy of the linear self-attention calculation result and thus ensuring the accuracy of the final self-attention output result.

[0022] Optionally, when the query matrix of the first segment includes a first query matrix and a second query matrix encoded using a positional encoding operator, and the key matrix of the first segment includes a first key matrix and a second key matrix encoded using a positional encoding operator, the implementation process of performing linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix may include: calculating the first self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the first query matrix, the absolute value matrices of the positive and negative matrices corresponding to the first key matrix, and the value matrix; calculating the second self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the second query matrix, the absolute value matrices of the positive and negative matrices corresponding to the second key matrix, and the value matrix; and determining the intra-segment self-attention of each element in the first segment based on the first self-attention and the second self-attention of each element in the first segment.

[0023] In this application, a dual-positional encoding operator is first used to positionally encode the original query matrix and the original key matrix of the segment, thereby obtaining a first query matrix, a first key matrix, a second query matrix, and a second key matrix. For the first query matrix and the first key matrix obtained by the first positional encoding operator, the aforementioned method of splitting positive and negative value elements can be used to perform linear self-attention calculation, thereby obtaining the first self-attention. Similarly, for the second query matrix and the second key matrix obtained by the second positional encoding operator, the aforementioned method of splitting positive and negative value elements can also be used to perform linear self-attention calculation, thereby obtaining the second self-attention. Based on this, the first and second self-attention are fused to obtain the self-attention calculation result of the segment, which includes the intra-segment self-attention of each element within the segment. Therefore, this application combines the method of encoding the query matrix and the key matrix using a dual-positional encoding operator with the method of splitting positive and negative value elements in the query matrix and the key matrix to perform linear self-attention calculation. This improves the accuracy of the linear self-attention calculation result, thereby improving the accuracy of the self-attention output result.

[0024] In a second aspect, a sequence processing apparatus is provided, the sequence processing apparatus comprising at least one module for performing the sequence processing method described in the first aspect above.

[0025] The at least one module may include a segmentation module, an intra-segment self-attention calculation module, an inter-segment self-attention calculation module, and a self-attention fusion module. The segmentation module divides the input sequence into multiple segments, each segment comprising multiple elements, with each segment comprising at least two elements, the multiple elements representing multiple data units in the data to be processed. The intra-segment self-attention calculation module calculates the self-attention between at least two elements within each segment, obtaining the intra-segment self-attention corresponding to each element. The inter-segment self-attention calculation module extracts feature information from the multiple segments and, based on the feature information, calculates the self-attention between each element in the input sequence and the multiple segments, obtaining the inter-segment self-attention corresponding to each element in the input sequence. The self-attention fusion module determines the self-attention output result based on the intra-segment and inter-segment self-attention corresponding to the multiple elements in the input sequence.

[0026] Optionally, the data to be processed is text, the data unit is a word in the text, and the multiple elements are feature vectors of multiple words in the text; or, the data to be processed is an image, the data unit is an image region in the image, and the multiple elements are feature vectors of multiple image regions in the image; or, the data to be processed is video, the data unit is a video unit in the video, and the multiple elements are feature vectors of multiple video units in the video; or, the data to be processed is speech data, the data unit is a speech segment in the speech data, and the multiple elements are feature vectors of multiple speech segments in the speech data.

[0027] Optionally, the feature information includes feature key vectors and feature value vectors; the inter-segment self-attention calculation module is specifically used to: extract features from the key matrix of the first segment among the plurality of segments to obtain the feature key vector of the first segment, wherein the key matrix of the first segment includes the key vectors of the elements within the first segment; and extract features from the value matrix of the first segment to obtain the feature value vector of the first segment, wherein the value matrix of the first segment includes the value vectors of the elements within the first segment.

[0028] Optionally, the inter-segment self-attention calculation module is specifically used to: perform self-attention calculation based on the query matrix of the input sequence, the feature key vectors and feature value vectors of the multiple segments, to obtain the inter-segment self-attention corresponding to each element in the input sequence, wherein the query matrix of the input sequence includes the query vectors of the elements in the input sequence.

[0029] Optionally, the intra-segment self-attention calculation module is specifically used to: perform linear self-attention calculation based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments, to obtain the intra-segment self-attention corresponding to each element in the first segment, wherein the query matrix of the first segment includes the query vector of the element in the first segment, the key matrix of the first segment includes the key vector of the element in the first segment, and the value matrix of the first segment includes the value vectors of at least two elements in the first segment.

[0030] Optionally, the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix. The intra-segment self-attention calculation module is further configured to: based on the position of each element in the first segment in the input sequence, use a first positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively to obtain the first query matrix and the first key matrix; based on the position of each element in the first segment in the input sequence, use a second positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively to obtain the second query matrix and the second key matrix.

[0031] Optionally, the intra-segment self-attention calculation module is specifically used to: obtain the positive and negative matrices corresponding to the query matrix of the first segment and the positive and negative matrices corresponding to the key matrix of the first segment, wherein the positive matrix includes positive elements in the corresponding matrix and the negative matrix includes negative elements in the corresponding matrix; determine the absolute value matrix of the negative matrix corresponding to the query matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the key matrix of the first segment, wherein the absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix; and perform linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix.

[0032] Optionally, when the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix, the intra-segment self-attention calculation module is specifically used to: calculate the first self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the first query matrix, the absolute value matrices of the positive and negative matrices corresponding to the first key matrix, and the value matrix; calculate the second self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the second query matrix, the absolute value matrices of the positive and negative matrices corresponding to the second key matrix, and the value matrix; and determine the intra-segment self-attention of each element in the first segment based on the first self-attention and the second self-attention of each element in the first segment.

[0033] Thirdly, a computing device is provided, the computing device including a processor, the processor being configured to execute at least one program instruction or code stored in a memory to implement the sequence processing method described in the first aspect above.

[0034] The computing device may further include the memory. Additionally, the processor may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), and one or more neural network processing units (NPUs). The CPU, GPU, and NPU can execute program instructions or code stored in the memory to jointly implement the sequence processing method described in the first aspect above.

[0035] Fourthly, a computing device cluster is provided, the computing device cluster including a plurality of computing devices, each computing device including a processor and a memory; the processor of the plurality of computing devices is used to execute instructions stored in the memory of the plurality of computing devices, so that the computing device cluster performs the sequence processing method described in the first aspect above.

[0036] The plurality of computing devices can execute the sequence processing method described in the first aspect in parallel. Alternatively, the plurality of computing devices can each execute different steps of the sequence processing method described in the first aspect. In this case, the plurality of computing devices can be connected via a network, through which intermediate results or data generated during the execution of the sequence processing method can be transmitted between the plurality of computing devices.

[0037] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computing device, cause the computing device to perform the sequence processing method described in the first aspect.

[0038] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computing device, causes the computing device to perform the sequence processing method described in the first aspect.

[0039] The technical effects achieved by the second to sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application;

[0043] Figure 4 A flowchart illustrating a sequence processing method provided in an embodiment of this application;

[0044] Figure 5 A flowchart illustrating linear self-attention computation based on a query matrix, key matrix, and value matrix of fragments, provided for embodiments of this application;

[0045] Figure 6 A schematic diagram illustrating an intra-segment self-attention calculation provided in an embodiment of this application;

[0046] Figure 7 A detailed schematic diagram of a sequence processing flow provided for an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the structure of a sequence processing device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0049] Before providing a detailed explanation of the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0050] Currently, the mainstream architectures of AI models such as LLM, CV models, and multimodal models are all based on the transformer structure. The transformer architecture, through its self-attention mechanism, enables efficient modeling of the relationship between any two elements in the input sequence. This allows the AI ​​model to fully consider the context of the input sequence when performing prediction tasks, thereby improving the accuracy of the prediction task.

[0051] It should be noted that the input sequence is used to represent the data to be processed, and an element in the input sequence represents a data unit in the data to be processed. For example, an element in the input sequence can be a feature value or feature vector of a data unit in the data to be processed. The data to be processed can be text, and the data unit can be a word in the text; or the data to be processed can be an image, and the data unit can be an image region in the image; or the data to be processed can be video, and the data unit can be a video unit in the video; or the data to be processed can be speech data, and the data unit can be a speech segment in the speech data, and so on.

[0052] Currently, AI models based on the transformer architecture determine the query (Q) matrix, key (K) matrix, and value (V) matrix based on the elements in the input sequence when calculating self-attention. Self-attention is then calculated based on these matrices. The Q matrix contains the query vectors of the elements in the input sequence, the K matrix contains the key vectors, and the V matrix contains the value vectors. Therefore, the size of each of these matrices depends on the number of elements in the input sequence. The more elements in the input sequence, the larger these matrices become, resulting in higher computational and memory overhead for the AI ​​model when calculating self-attention.

[0053] Therefore, to reduce the computational cost and memory overhead of AI models when calculating self-attention, current methods in training and inference scenarios, for long input sequences containing many elements, involve segmenting the input sequence into multiple segments and calculating the self-attention between elements within each segment to obtain the corresponding self-attention output. However, because the input sequence is segmented and self-attention is calculated at the segment level, the self-attention output only contains local contextual information of elements within each segment, resulting in low accuracy and consequently, insufficient accuracy in prediction tasks based on this self-attention output.

[0054] To address the problems existing in related technologies, embodiments of this application provide a sequence processing method. In this method, for an input sequence comprising multiple segments, self-attention between elements within each segment can be calculated, thereby obtaining intra-segment self-attention containing local context information. Furthermore, feature information from multiple segments can be extracted and used to calculate self-attention between each element and multiple segments, thereby obtaining inter-segment self-attention containing global context information. Based on this, the self-attention output result obtained from the intra-segment and inter-segment self-attention of multiple elements will simultaneously contain both the local context information of each element within the segment and the global context information of each element. Therefore, the sequence processing method provided by embodiments of this application improves the accuracy of the self-attention output result while minimizing the computational and memory overhead of self-attention calculation, thereby enhancing the accuracy of the output sequence predicted based on the self-attention output result.

[0055] The sequence processing method provided in this application embodiment can be executed by a computing device, which can be a server or a terminal device. For example, Figure 1 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 1 As shown, the computing device may include a processor 101, a memory 102, a communication bus 103, and a communication interface 104. It should be noted that... Figure 1 The illustrated device structure does not constitute a limitation on the computing device. A computing device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Furthermore, the embodiments of this application do not limit the number of processors 101 and memory 102 included in the computing device 100. The following is in conjunction with… Figure 1 A detailed description of each component of a computing device:

[0056] Processor 101 is the control and computing center of the computing device. Processor 101 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), neural network processing unit (NPU), microprocessor (MP), digital signal processor (DSP).

[0057] Memory 102 may be high bandwidth memory (HBM), read-only memory (ROM), or other types of static storage devices capable of storing static information and instructions; random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computing device, but not limited thereto. Memory 102 may exist independently and be connected to processor 101 via communication bus 103. Alternatively, memory 102 may be integrated with processor 101.

[0058] It should be noted that the memory 102 stores executable program code, and the processor 101 executes the executable program code to perform the sequence processing method in the following embodiments. That is, the memory 102 stores instructions for performing the following sequence processing method.

[0059] In one example, the CPU included in processor 101 can execute executable program code in memory 102 to implement the sequence processing method in the following embodiments.

[0060] In another example, different parts of the executable program code in memory 102 may be executed by the CPU, GPU, and NPU included in processor 101 to collaboratively implement the sequence processing method in the following embodiments.

[0061] For example, the CPU can execute a portion of the program code to implement the operations described in the sequence processing method below, such as segmenting the input sequence, obtaining the Q matrix, K matrix, and V matrix of each segment, and sending the obtained matrices to the GPU. Correspondingly, the GPU can perform matrix operations based on the matrices sent by the CPU to implement self-attention calculation.

[0062] Optionally, memory 102 can be used to store data generated during the execution of executable program code by processor 101 to implement the sequence processing method. For example, memory 102 can store the Q matrix, K matrix, V matrix of each segment, as well as intermediate and final results obtained by operations on the aforementioned matrices.

[0063] The communication bus 103 may include a path for transmitting information between the various components of the computing device 100. The communication bus 103 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0064] The communication interface 104 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0065] As one embodiment, the computing device may further include output devices and input devices. The output device communicates with the processor 101 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 101 and can receive input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0066] The sequence processing method provided in this application embodiment can also be executed by a computing device cluster including multiple computing devices. The computing devices in this cluster can be servers, such as central servers, edge servers, or local servers in a local data center. In some embodiments, the computing devices can also be terminal devices such as desktop computers, laptops, or smartphones.

[0067] For example, see Figure 2 The computing device cluster 200 includes multiple such Figure 1 The computing device 100 shown.

[0068] In some possible implementations, the memory 102 in each computing device 100 may store the same instructions for executing the sequence processing method. In this way, multiple computing devices 100 can execute the sequence processing method provided in the embodiments of this application in parallel.

[0069] For example, in one possible application scenario, a computing device cluster includes multiple computing devices 100, each deploying the same AI model to be trained. Each computing device can train its deployed AI model using different training samples. This AI model can be an encoder-only model, a decoder-only model, or an encoder-decoder model. In this case, during training, one or more self-attention layers in the AI ​​model of each computing device 100 can be computed using the sequence processing method provided in this application embodiment.

[0070] In other possible implementations, the memories 102 of the multiple computing devices 100 in the computing device cluster may each store partial instructions for executing the sequence processing method in the embodiments described below. In other words, a combination of multiple computing devices 100 can jointly implement the sequence processing method provided in the embodiments of this application. In this case, see [link to relevant documentation]. Figure 3 The multiple computing devices 100 ( Figure 3 The diagram illustrates two computing devices 100 that can be connected via a network to transmit intermediate results generated during the execution of the sequence processing method. The network can be a wide area network (WAN) or a local area network (LAN), etc.

[0071] The sequence processing method provided in the embodiments of this application will be described in detail below.

[0072] Figure 4 This is a flowchart illustrating a sequence processing method provided in an embodiment of this application. The detailed implementation process of this method will be described below, taking the execution of this method by a computing device as an example. See also... Figure 4 The method includes the following steps S401 to S404, and S401 to S405 will be described in detail below.

[0073] S401: Divide the input sequence into multiple segments.

[0074] In this embodiment, the input sequence can refer to a sequence representing the data to be processed. The input sequence includes multiple elements arranged in order, each element representing a data unit within the data to be processed. Each element corresponds one-to-one with a data unit; that is, one element can represent one data unit. In this embodiment, the data units within the data to be processed can also be called tokens, and the multiple data units can have a specific order. Depending on the application scenario, the data to be processed can be of various types.

[0075] For example, the data to be processed can be text, and the data units in the data to be processed can refer to words in the text. A word can refer to the smallest semantic unit in the text; for example, a phrase or a character may be a word.

[0076] When the data to be processed is text, in one possible implementation, multiple elements in the input sequence can refer to feature values ​​or feature vectors of multiple words in the text.

[0077] In one example, a computing device can segment text into multiple tokens, and then represent each token with a numerical value, thereby obtaining a digitized sequence corresponding to the text. The numerical value used to represent each token is called its feature value. The input sequence can refer to the digitized sequence corresponding to the text, and correspondingly, the elements in the input sequence can refer to the feature values ​​in that digitized sequence.

[0078] In another example, after obtaining the digitized sequence corresponding to the text using the method described above, the computing device generates a feature vector representing each word based on the feature value of each word in the digitized sequence, thus obtaining a sequence containing multiple feature vectors. For example, this sequence can be represented as a feature matrix. Each row of the feature matrix represents a feature vector. The input sequence can refer to a sequence containing these multiple feature vectors, and correspondingly, the elements in the input sequence can refer to the feature vectors in that sequence.

[0079] In another possible implementation, the multiple elements in the input sequence can also refer to the intermediate encoding vectors of multiple words in the text. For example, after obtaining a sequence containing the feature vectors of each word in the text using the above method, the computing device can process the sequence through at least one connected transformer layer to output a processing result. Each transformer layer includes a self-attention layer and a fully connected layer. The self-attention layer can perform self-attention calculation on the input sequence using the method provided in this embodiment or a traditional self-attention calculation method, and output a self-attention output result. The fully connected layer can be used to perform a linear transformation on the self-attention output result to output a processing result containing the intermediate encoding vector of each word, and this processing result will be used as the input to the next transformer layer, and so on. Therefore, the input sequence in this embodiment can refer to the processing result output by a transformer layer containing the intermediate encoding vectors of multiple words in the text.

[0080] Optionally, the data to be processed can also be an image, and the data unit in the data to be processed can refer to an image region in the image. For example, for an image frame to be processed, the image can be divided into multiple rectangular image regions of the same size, and each image region is a data unit.

[0081] When the data to be processed is an image, in one possible implementation, the multiple elements in the input sequence can refer to feature vectors of multiple image regions in the image. For example, for each image region in the image, the computing device can perform feature extraction on the image region; for instance, feature information of the image region can be extracted through convolution operations to obtain a feature vector of the image region. Then, the feature vectors of the multiple image regions are sorted according to their positions in the image to obtain a sequence containing multiple feature vectors, and this sequence is used as the input sequence.

[0082] In another possible implementation, the multiple elements in the input sequence can also refer to intermediate encoded vectors of multiple image regions in the image. For example, after obtaining a sequence containing feature vectors of multiple image regions in the image, the computing device can also process this sequence through at least one connected transformer layer to output a processing result. This processing result contains intermediate encoded vectors of multiple image regions. Therefore, the input sequence in this embodiment can also refer to the processing result output by a transformer layer containing intermediate encoded vectors of multiple image regions in the image.

[0083] Optionally, the data to be processed can also be video, and the data unit in the data to be processed can refer to a video unit in the video. Each video unit can include one or more video frames. Where a video unit includes multiple video frames, these multiple video frames can be consecutive.

[0084] When the data to be processed is video, in one possible implementation, the multiple elements in the input sequence can refer to feature vectors of multiple video units in the video. For example, for each video unit in the video, the computing device can perform feature extraction on the video unit to obtain a feature vector representing the video unit. Then, the feature vectors of these multiple video units are sorted according to their chronological order to obtain a sequence containing multiple feature vectors, which is used as the input sequence.

[0085] In another possible implementation, the multiple elements in the input sequence can also refer to the intermediate encoded vectors of multiple video units in the video. For example, after obtaining a sequence containing feature vectors of multiple video units, the computing device can also process the sequence through at least one connected transformer layer to output a processing result. This processing result contains the intermediate encoded vectors of multiple video units. Therefore, the input sequence in this embodiment can also refer to the processing result output by a transformer layer containing the intermediate encoded vectors of multiple video units in the video.

[0086] Optionally, the data to be processed can also be speech data, and the data units in the data to be processed can be speech segments from the speech data. Speech segments can be segments obtained by dividing the speech data according to chronological order.

[0087] When the data to be processed is speech data, in one possible implementation, the multiple elements in the input sequence can refer to feature vectors of multiple speech segments in the speech data. For example, the computing device can extract features from each speech segment in the speech data to obtain a feature vector representing each speech segment. Then, the feature vectors of the multiple speech segments are sorted according to their chronological order to obtain a sequence containing multiple feature vectors, which is used as the input sequence.

[0088] In another possible implementation, the multiple elements in the input sequence can also refer to intermediate encoding vectors of multiple speech segments in the speech data. For example, after obtaining a sequence containing feature vectors of multiple speech segments, the computing device can also process the sequence through at least one connected transformer layer to output a processing result. This processing result contains intermediate encoding vectors of multiple speech segments. Therefore, the input sequence in this embodiment can also refer to the processing result output by a transformer layer containing intermediate encoding vectors of multiple speech segments in the speech data.

[0089] The above text illustrates four types of data to be processed, but the types of data to be processed are not limited to those described above. For example, the data to be processed may be other types of time-series data, such as health monitoring data of a target object. Alternatively, the data to be processed may also be multimodal data including at least two of the following: text, images, video, and audio.

[0090] It should also be noted that when the method provided in this application embodiment is applied to an AI model in a training scenario, the data to be processed can refer to training samples, test samples, or validation samples to be processed. When the method provided in this application embodiment is applied to an AI model in an inference scenario, the data to be processed can refer to application data to be processed.

[0091] For an input sequence corresponding to the data to be processed, the computing device can divide the input sequence into multiple segments, wherein each segment includes at least two elements, and the at least two elements are consecutive elements in the input sequence.

[0092] For example, the computing device can segment the input sequence according to preset segmentation parameters. For instance, the preset segmentation parameters include a segment size, where the segment size refers to the number of elements *m* contained in a segment, where *m* is an integer not less than 2. Based on this, the computing device can divide every *m* consecutive elements in the input sequence into a segment according to the segment size, starting from the first element, thereby obtaining multiple segments. The total number of elements in the input sequence may not be an integer multiple of *m*. Thus, during the segmentation process, there may be fewer than *m* elements remaining. In this case, if the number of remaining elements is not less than 2, the remaining elements can be used as the last segment; if the number of remaining elements is 1, the remaining elements can be added to the last segment. In this case, the last segment contains (m+1) elements.

[0093] For example, assuming the input sequence contains 4000 elements and the segment size is 100, the input sequence can be divided into 40 segments, each containing 100 consecutive elements.

[0094] Alternatively, in one possible implementation, the computing device may start from the first element of the input sequence, add a first marker element before the first element of every consecutive m elements to mark the beginning of a new segment, and add a last marker element after the last element of every consecutive m elements to mark the end of a segment, thus obtaining a segment. In this case, a segment may include (m+2) elements.

[0095] S402: Calculate the self-attention between elements within each segment to obtain the segment self-attention corresponding to each element.

[0096] After dividing the input sequence into multiple segments, the computing device can determine the original Q matrix, original K matrix, and V matrix for each segment. Then, based on the Q matrix, K matrix, and V matrix of each segment, self-attention calculation is performed to obtain the self-attention calculation result for the corresponding segment.

[0097] The Q-matrix and K-matrix of each segment can be the original Q-matrix and original K-matrix of the corresponding segment, or the Q-matrix of each segment can be a matrix obtained by positionally encoding the original Q-matrix of the corresponding segment, and the K-matrix of each segment can be a matrix obtained by positionally encoding the original K-matrix of the corresponding segment. Furthermore, the self-attention calculation result for each segment includes the intra-segment self-attention corresponding to each element within the corresponding segment.

[0098] For example, taking the first segment of a plurality of segments as an example, if the multiple elements in the input sequence are feature vectors used to characterize multiple data units in the data to be processed, that is, the input sequence is a sequence containing multiple feature vectors, then the elements contained in the first segment can form a feature matrix in chronological order. Each row in this feature matrix represents one element, i.e., one feature vector, and the order of the rows corresponds to the order of the elements. Based on this, the computing device can multiply the feature matrix of the first segment by a first projection matrix to obtain the original Q matrix of the first segment; multiply the feature matrix of the first segment by a second projection matrix to obtain the original K matrix of the first segment; and multiply the feature matrix of the first segment by a third projection matrix to obtain the V matrix of the first segment.

[0099] The first, second, and third projection matrices are three different weight parameter matrices. In the training scenario, these three projection matrices represent the parameters to be learned; in the inference scenario, they represent three pre-learned parameters.

[0100] Furthermore, the original Q matrix of the first segment includes the query vector for each element in the first segment, where each row of the original Q matrix represents the query vector of one element, and the order of the rows corresponds to the order of the elements. Similarly, the original K matrix of the first segment includes the key vector for each element in the first segment, where each row of the original K matrix represents the key vector of one element, and the order of the rows corresponds to the order of the elements. The V matrix of the first segment includes the value vector for each element in the first segment, where each row of the V matrix represents the value vector of one element, and the order of the rows corresponds to the order of the elements. In addition, the original Q matrix, the original K matrix, and the V matrix have the same number of columns, meaning that the dimensions of the query vector, key vector, and value vector for each element are the same.

[0101] Optionally, if multiple elements in the input sequence are feature values ​​representing multiple data units in the data to be processed, that is, if the input sequence is a sequence containing multiple feature values, then each element in the first segment can be represented by a feature vector, thus obtaining multiple feature vectors. These multiple feature vectors can be arranged into a feature matrix according to the order of the elements. This feature matrix is ​​the feature matrix of the first segment. Based on this, the computing device can refer to the above method to multiply the feature matrix of the first segment by different projection matrices to obtain the original Q matrix, the original K matrix, and the V matrix of the first segment.

[0102] For each segment in the input sequence, the original Q matrix, original K matrix, and V matrix of the corresponding segment can be obtained using the method described above. Then, the computing device can perform self-attention calculations in parallel based on the original Q matrix, original K matrix, and V matrix of each segment, thereby obtaining the self-attention calculation result for each segment.

[0103] Next, we will take the first segment out of multiple segments as an example to explain in detail how to implement self-attention between elements within a segment.

[0104] In the first implementation, the computing device can perform traditional self-attention calculation based on the Q matrix, K matrix, and V matrix of the first segment to obtain the self-attention calculation result of the first segment. Here, the Q matrix of the first segment refers to the original Q matrix of the first segment, and the K matrix of the first segment refers to the original K matrix of the first segment.

[0105] For example, the computing device can calculate the attention weight matrix based on the Q matrix and K matrix of the first segment, and then calculate the self-attention calculation result of the first segment based on the attention weight matrix and the V matrix of the first segment.

[0106] Specifically, the computing device can calculate the product of the Q matrix and the transpose of the K matrix to obtain the attention score matrix. Each element in this attention score matrix is ​​the attention score between two elements in the first segment, representing the similarity between the two elements. Then, the softmax function is used to normalize the attention scores in this matrix, making the attention scores between highly similar elements more prominent, thus obtaining the attention weight matrix. Each element in the attention weight matrix is ​​the attention weight between two elements in the first segment, representing the mutual influence between the two elements; the larger the attention weight, the higher the correlation between the two elements. Next, the computing device can calculate the product of this attention weight matrix and the V matrix of the first segment to obtain the self-attention calculation result for the first segment. This self-attention calculation result is a matrix of the same size as the V matrix, where each row of the matrix corresponds to the intra-segment self-attention of an element in the first segment.

[0107] For each of the multiple segments, the computing device can calculate the self-attention calculation result of each segment by referring to the above method.

[0108] In the second implementation, the computing device can perform linear self-attention calculation based on the Q matrix, K matrix and V matrix of the first segment, thereby obtaining the self-attention calculation result of the first segment.

[0109] In one possible case, the Q matrix and K matrix of the first segment refer to the original Q matrix and the original K matrix of the first segment.

[0110] In another possible scenario, the Q matrix of the first segment may include a first Q matrix and a second Q matrix obtained by positional encoding of the original Q matrix of the first segment, and the K matrix of the first segment may include a first K matrix and a second K matrix obtained by positional encoding of the original K matrix of the first segment.

[0111] For example, the computing device can perform position encoding on the original Q matrix and the original K matrix of the first segment based on the position of each element in the first segment in the input sequence, using a first position encoding operator, to obtain a first Q matrix and a first K matrix; and perform position encoding on the original Q matrix and the original K matrix of the first segment based on the position of each element in the first segment in the input sequence, using a second position encoding operator, to obtain a second query matrix and a second key matrix.

[0112] Here, the position of each element in the input sequence can refer to its index within the sequence. For example, if there are N elements in the input sequence, and a certain element is the i-th element, then its position can be represented by i. Furthermore, the first and second positional encoding operators are different operators. For instance, the first positional encoding operator can be an increasing function with respect to the element's position, and the second positional encoding operator can be a decreasing function with respect to the element's position. Alternatively, the first and second positional encoding operators can be two functions designed based on concavity / convexity. For example, the first positional encoding operator can be a concave function with respect to the element's position, and the second positional encoding operator can be a convex function with respect to the element's position.

[0113] For example, the first position encoding operator can be the function shown in equation (1) below, and the second position encoding operator can be the function shown in equation (2) below.

[0114]

[0115] Among them, when encoding the original Q matrix, x i Let x be a query vector in the original Q matrix, and let x be the i-th element in the input sequence. When encoding the original K matrix, x i Let x be a key vector in the original K matrix, and let the element corresponding to the key vector be the i-th element in the input sequence, where i is x. i The corresponding element is located in the input sequence, where N is the total number of elements in the input sequence.

[0116] The two functions given above are merely examples of two position encoding operators. Alternatively, the computing device may also use other forms of functions for position encoding. For example, the computing device may use a rotation position encoding function to perform position encoding on the original Q matrix and the original K matrix of the first segment based on the position of each element in the first segment in the input sequence.

[0117] In this embodiment, based on the position of each element in the first segment within the input sequence, a positional encoding operator is used to positionally encode the original Q-matrix and original K-matrix of the first segment. This allows the encoded Q-matrix and K-matrix to contain the absolute positional information of each element in the first segment. Based on this, when calculating self-attention using the encoded Q-matrix and K-matrix, the correlation between elements can be captured more accurately; that is, a more precise correlation or similarity between elements can be obtained, thereby improving the accuracy of the self-attention calculation results.

[0118] After obtaining the Q, K, and V matrices of the first segment, the computing device can perform linear self-attention computation based on the Q, K, and V matrices of the first segment.

[0119] It should be noted that, as described in the first implementation above regarding traditional self-attention computation, the attention score matrix is ​​first obtained by multiplying the transposes of the Q and K matrices. Then, the attention score matrix is ​​normalized and multiplied by the V matrix to obtain the self-attention result. Unlike traditional self-attention computation, linear self-attention computation utilizes the commutative property of matrix multiplication. It first calculates the product of the transpose of the K matrix and the V matrix, and then calculates the product of the Q matrix and this product, thus achieving lower matrix operation complexity. Furthermore, considering that traditional self-attention computation involves normalizing the attention score matrix using a softmax function to highlight the attention scores of elements with high similarity, linear self-attention computation introduces a kernel function to process the Q and K matrices before performing the matrix multiplication operation. This approximates the function of the softmax function, ensuring that the final calculated self-attention value closely approximates the result of traditional self-attention computation.

[0120] Specifically, in one possible implementation, see [link to relevant documentation]. Figure 5 The implementation process of linear self-attention computation based on the Q matrix, K matrix, and V matrix of the first segment can include S4021 to S4023. S4021 to S4023 will be described below.

[0121] S4021: Obtain the positive and negative matrices corresponding to the Q matrix of the first segment and the positive and negative matrices corresponding to the K matrix of the first segment, wherein the positive matrix includes the positive elements in the corresponding matrix and the negative matrix includes the negative elements in the corresponding matrix.

[0122] In the first case, the Q matrix and K matrix of the first segment refer to the original Q matrix and the original K matrix of the first segment, respectively. For ease of description, the original Q matrix is ​​denoted by Q, and the original K matrix is ​​denoted by K. In this case, the computing device can split the positive and negative elements in the Q matrix of the first segment to obtain the corresponding positive matrix Q. + and negative matrix Q - In this context, positive elements are those with values ​​greater than 0, negative elements are those with values ​​less than 0, and Q equals the corresponding Q. + and Q - Similarly, the computing device can split the positive and negative elements in the first segment K to obtain the positive matrix K corresponding to K. +and negative matrix K - Where K equals the corresponding K + and K - The sum of.

[0123] In the second case, the Q matrix of the first segment includes a first Q matrix and a second Q matrix obtained from position encoding, and the K matrix of the first segment includes a first K matrix and a second K matrix obtained from position encoding. For ease of description, the first Q matrix is ​​denoted by Q1, the second Q matrix by Q2, the first K matrix by K1, and the second K matrix by K2. In this case, the computing device can refer to the above method to split the positive and negative elements in Q1, Q2, K1, and K2 respectively, thereby obtaining the positive matrix corresponding to Q1. and negative matrix The positive matrix corresponding to Q2 and negative matrix The positive matrix corresponding to K1 and negative matrix The positive matrix corresponding to K2 and negative matrix

[0124] S4022: Determine the absolute value matrix of the negative matrix corresponding to the Q matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the K matrix of the first segment. The absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix.

[0125] For the negative matrices corresponding to the Q and K matrices of the first segment, the computing device can take the absolute value of the negative elements in each negative matrix to obtain the corresponding absolute value matrix.

[0126] Regarding the first scenario in S4021, the computing device can process Q. - and K - The absolute values ​​of the negative elements in the matrix are taken as absolute values, thus obtaining the absolute value matrix |Q. - | and | K - |

[0127] Regarding the second scenario in S4021, the computing device can respectively... The absolute values ​​of the negative elements in the matrix are taken as absolute values, thus obtaining the absolute value matrix.

[0128] S4023: Perform linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, and the absolute value matrices and value matrices of the positive and negative matrices corresponding to the key matrix of the first segment.

[0129] For the first case in S4021, the computing device can be based on Q. + K +、|Q - |、|K - | Perform self-attention calculation to obtain the self-attention calculation result of the first segment.

[0130] For example, a computing device can respond to Q + and K + Each matrix element in the array is raised to the power of p, and the result of the power operation (Q) is used to... + ) p and (K) + ) p As input to the kernel function, this kernel function is used to apply (Q) + ) p and (K) + ) p The positive Q matrix and the positive K matrix are processed separately to obtain the processed positive Q matrix and the processed positive K matrix. Then, the transpose of the processed positive K matrix is ​​multiplied by the V matrix of the first segment to obtain the first product. The processed positive Q matrix is ​​then multiplied by the first product to obtain the first part of self-attention.

[0131] In addition, computing devices can process |Q separately. - | and | K - | Perform p-th power operations on the matrix elements in | and use the result of the power operation |Q - | p and |K - | p As input to the kernel function, this kernel function is used to apply |Q - | p and |K - | p The process involves processing to obtain the processed Q-absolute value matrix and the processed K-absolute value matrix. Then, the processed K-absolute value matrix is ​​multiplied by the V matrix of the first segment to obtain the second product. The processed Q-absolute value matrix is ​​then multiplied by the second product to obtain the second part of the self-attention.

[0132] The computing device can sum the first part of attention and the second part of self-attention to obtain the self-attention calculation result of the first segment.

[0133] Regarding the second scenario in S4021, the computing device can be based on Given the V matrix of the first segment, calculate the first self-attention for each element within the first segment; based on... Given the V matrix of the first segment, calculate the second self-attention for each element within the first segment; based on the first and second self-attentions for each element within the first segment, determine the intra-segment self-attention for each element within the first segment.

[0134] For example, a computing device may refer to the manner described above to... Each matrix element in the array is raised to the power of p, and the result of the power operation is used as the input to a kernel function. The kernel function processes the result of the power operation to obtain the processed result. and after processing After that, the processed Multiplying the transpose of the matrix by the V matrix of the first segment yields the third product. The processed product... Multiplying this by the third product yields the third part of self-attention. and Each matrix element in the array is raised to the power of p, and the result of the power operation is used as the input to a kernel function. The kernel function processes the result of the power operation to obtain the processed result. and after processing After that, the processed Multiplying the transpose of the matrix by the V matrix of the first segment yields the fourth product. The processed product... Multiply by the fourth product to obtain the fourth part of self-attention. Summing the third and fourth parts of self-attention yields the first self-attention calculation result, which includes the first self-attention of each element within the first segment.

[0135] Referring to the method described above for calculating the first self-attention, the computing device can be based on The V matrix of the first segment is used to obtain the second self-attention calculation result, which includes the second self-attention of each element in the first segment.

[0136] After obtaining the first and second self-attention calculation results, the computing device can sum them. That is, the computing device can sum the first and second self-attention of each element to obtain the self-attention calculation result for the first segment. This first segment's self-attention calculation result includes the intra-segment self-attention of each element within the first segment, and the intra-segment self-attention of each element is equal to the sum of the first and second self-attention of the corresponding element.

[0137] Optionally, after summing the first self-attention calculation result and the second self-attention calculation result, the computing device can also use the softmax function to normalize the summing result to obtain the self-attention calculation result of the first segment.

[0138] It should be noted that the kernel function used to process the positive matrix and the absolute value matrix can be a function based on the linear rectification function (ReLU). Currently, in related technologies, when performing linear self-attention calculations, if the kernel function is based on ReLU, it is usually used directly to process the Q matrix and K matrix. However, based on the characteristics of ReLU, when the input value is negative, the ReLU output is 0, and when the input value is positive, the ReLU output value equals the input value. Therefore, after directly processing the Q matrix and K matrix using the kernel function, the information represented by the negative elements in the Q matrix and K matrix will be lost. Consequently, the accuracy of the self-attention calculated based on the processed Q matrix and K matrix decreases significantly. In this embodiment, the Q matrix and K matrix are split into positive and negative matrices respectively, and the absolute values ​​of the elements in the negative matrix are taken. Then, the kernel function is used to process the positive matrix and the absolute value matrix. In this way, the information represented by the negative elements in the Q matrix and K matrix is ​​preserved, thereby improving the accuracy of the self-attention calculated subsequently.

[0139] The above describes one implementation of intra-segment self-attention calculation using a linear self-attention mechanism, as provided in this application. In other possible implementations, the computing device may not split the positive and negative elements in the Q and K matrices of the first segment. Instead, it may directly process the Q and K matrices using a kernel function, calculate the product of the transpose of the processed K matrix and the V matrix, and then multiply the processed Q matrix by this product to obtain the self-attention calculation result for the first segment. Here, the Q and K matrices of the first segment can be the original Q matrix and the original K matrix, respectively.

[0140] Optionally, the Q matrix of the first segment may also include a first Q matrix and a second Q matrix obtained through the aforementioned positional encoding, and the K matrix of the first segment may include a first K matrix and a second K matrix obtained through the aforementioned positional encoding. In this case, the computing device can directly process the first Q matrix and the first K matrix using a kernel function, calculate the product of the transpose of the processed first K matrix and the V matrix, and multiply the processed first Q matrix by this product to obtain the first self-attention calculation result. The second Q matrix and the second K matrix are then processed using a kernel function, the product of the transpose of the processed second K matrix and the V matrix is ​​calculated, and the processed second Q matrix is ​​multiplied by this product to obtain the second self-attention calculation result. Finally, the first self-attention calculation result and the second self-attention calculation result are summed to obtain the self-attention calculation result of the first segment.

[0141] Figure 6This is a schematic diagram illustrating an embodiment of self-attention within a computational segment in this application. For example... Figure 6 As shown, the original Q matrix Q and the original K matrix K of the first segment are positionally encoded using positional encoding operator f1 to obtain Q1 and K1, respectively, and then positionally encoded using positional encoding operator f2 to obtain Q2 and K2. Next, linear self-attention calculation is performed based on Q1, K1, and the V matrix V to obtain the first self-attention calculation result O1; and based on Q2, K2, and V, linear self-attention calculation is performed to obtain the second self-attention calculation result O2. The sum of O1 and O2 yields the intra-segment self-attention calculation result O of the first segment.

[0142] For each segment in the input sequence, the computing device can refer to the above method to calculate the self-attention between elements within the segment. Thus, the intra-segment self-attention corresponding to each element in the input sequence can be obtained. Since the intra-segment self-attention of each element is the self-attention between the corresponding element and elements within its own segment, the intra-segment self-attention of each element contains the local context information within the corresponding element's segment, but does not contain the global context information outside the segment where the corresponding element is located. Based on this, in this embodiment, the computing device can also extract the feature information of each segment through S403 to S405, and use the feature information of each segment to calculate the self-attention between elements and segments, thereby capturing the global context information of the elements.

[0143] S403: Extract feature information from multiple segments.

[0144] In the embodiments of this application, the computing device can extract feature information of each segment, wherein the feature information of each segment can be used to characterize the content of the corresponding segment.

[0145] In some embodiments, the computing device can perform feature extraction on the K matrix of each segment to obtain the feature key vector of the corresponding segment, and perform feature extraction on the V matrix of each segment to obtain the feature value vector of the corresponding segment. The feature information of each segment includes the feature key vector and the feature value vector of the corresponding segment.

[0146] It should be noted that the K matrix of each segment can refer to the original K matrix obtained by multiplying the feature matrix of each segment by the second projection matrix, and the V matrix of each segment can refer to the original V matrix obtained by multiplying the feature matrix of each segment by the third projection matrix.

[0147] For example, taking the first segment as an example, the computing device can use a pooling algorithm to extract features from the K matrix and V matrix of the first segment, thereby obtaining a feature key vector and an eigenvalue vector with the same number of columns as the K matrix and V matrix of the first segment. The pooling algorithm can be either max pooling or average pooling; this embodiment of the application does not limit the specific algorithm used.

[0148] In other embodiments, the computing device may also perform feature extraction on the feature matrix of each segment to obtain the feature vector of each segment, and then multiply the feature vector of each segment by the second projection matrix and the third projection matrix respectively to obtain the feature key vector and feature value vector of each segment.

[0149] S404: Based on the feature information of multiple segments, calculate the self-attention between each element in the input sequence and multiple segments to obtain the inter-segment self-attention corresponding to each element in the input sequence.

[0150] After obtaining the feature information of multiple segments, the computing device can perform self-attention calculation based on the Q matrix of the input sequence, the feature key vectors and eigenvalue vectors of multiple segments, thereby obtaining the inter-segment self-attention corresponding to each element in the input sequence. The query matrix of the input sequence includes the query vector of the elements in the input sequence.

[0151] For example, the computing device can merge the original Q matrices of multiple segments according to the order of the segments to obtain the Q matrix of the input sequence. The feature key vectors of the multiple segments are arranged according to the order of the segments to obtain a segment K matrix, where each row of the segment K matrix is ​​a feature key vector of one segment, and the order of the rows represents the order of the segments. The feature value vectors of the multiple segments are arranged according to the order of the segments to obtain a segment V matrix, where each row of the segment V matrix is ​​a feature value vector of one segment, and the order of the rows represents the order of the segments.

[0152] In one possible implementation, the computing device can perform conventional self-attention computation based on the Q matrix, segment K matrix, and segment V matrix of the input sequence to obtain the inter-segment self-attention computation result. This inter-segment self-attention computation result includes the inter-segment self-attention for each element in the input sequence.

[0153] It should be noted that the implementation method of traditional self-attention computation based on the Q matrix, K matrix and V matrix of the input sequence can refer to the implementation process of traditional self-attention computation based on the Q matrix, K matrix and V matrix of the segments introduced in S402, which will not be repeated here.

[0154] In another possible implementation, the computing device can perform linear self-attention calculation based on the Q matrix, segment K matrix, and segment V matrix of the input sequence to obtain the inter-segment self-attention calculation result. The specific calculation method can be found in the implementation of linear self-attention calculation based on the segment-based Q matrix, K matrix, and V matrix described in S402, and will not be repeated here.

[0155] S405: Determine the self-attention output result based on the intra-segment self-attention and inter-segment self-attention corresponding to multiple elements in the input sequence.

[0156] Through S402, the computing device can obtain the intra-segment self-attention corresponding to each element in the input sequence. Through S403 and S404, the computing device can obtain the inter-segment self-attention corresponding to each element in the input sequence. Based on this, the computing device can fuse the intra-segment self-attention and inter-segment self-attention corresponding to each element to obtain the self-attention output result.

[0157] For example, the computing device can merge the self-attention calculation results of multiple segments in the input sequence according to the order of the segments to obtain the intra-segment self-attention calculation result of the input sequence. This intra-segment self-attention calculation result is then summed with the inter-segment self-attention calculation result to obtain the self-attention output result. Alternatively, the computing device can multiply the intra-segment self-attention calculation result by a first preset weight to obtain a first result, multiply the inter-segment self-attention calculation result by a second preset weight to obtain a second result, and then sum the first and second results to obtain the self-attention output result. The sum of the first and second preset weights is equal to 1.

[0158] Based on the sequence processing methods described above, Figure 7 This is a detailed schematic diagram illustrating a sequence processing flow according to an embodiment of this application. See also... Figure 7 The input sequence X is divided into n segments. For the i-th segment, the Q-matrix Q of the i-th segment is obtained based on the feature matrix of the i-th segment. i K matrix K i and matrix V i Where i takes values ​​from 1 to n. Then, using the Q, K, and V matrices of each segment, the self-attention between elements within each segment is calculated, thus obtaining the self-attention calculation result for each segment. For example, the self-attention calculation result for the first segment is O(n). 1 The self-attention calculation result for the second segment is O. 2 And so on. The self-attention calculation results of n segments are combined in the order of the segments to form the intra-segment self-attention calculation result O of the input sequence. X-1The self-attention calculation result for each segment includes the intra-segment self-attention of the elements within that segment. Thus, O X-1 Including intra-segment self-attention for each element in the input sequence.

[0159] In addition, for the i-th segment out of n segments, K i Feature extraction is performed to obtain the feature key vector Ek. i The feature key vectors of n segments form the segment K matrix K'; for the V of the i-th segment i Feature extraction is performed to obtain the feature vector Ev. i The eigenvalue vectors of n segments form the segment K matrix V'. The Q matrices of the n segments are then arranged in chronological order to form the Q matrix Q of the input sequence. X Based on Q X 、K'、V i The inter-segment self-attention calculation result O of the input sequence is obtained. X-2 Among them, O X-2 This includes inter-segment self-attention for each element in the input sequence. (The O...) X-1 and O X-2 Sum, we get O X O X This is the output result of self-attention.

[0160] In this embodiment, for an input sequence comprising multiple segments, self-attention between elements within each segment can be calculated to obtain intra-segment self-attention containing local context information. Furthermore, feature information from multiple segments can be extracted and used to calculate self-attention between each element and multiple segments, resulting in inter-segment self-attention containing global context information. Based on this, the self-attention output obtained from intra-segment and inter-segment self-attention of multiple elements will simultaneously contain the local context information of each element within the segment and the global context information outside the segment where each element resides. Therefore, the sequence processing method provided in this embodiment improves the accuracy of the self-attention output while minimizing the computational and memory overhead of self-attention calculation, thereby improving the accuracy of the output sequence predicted based on the self-attention output.

[0161] In addition, in the embodiments of this application, a linear self-attention algorithm can be used to calculate the self-attention between elements within each segment and the self-attention between each element in the input sequence and multiple segments, thereby further reducing the amount of computation and memory overhead.

[0162] When performing linear self-attention calculations, the positive and negative elements in the Q and K matrices can be split into two matrices. The absolute values ​​of the elements in the negative matrix containing negative elements are then taken. Subsequently, a kernel function is used to process the positive matrix containing positive elements and the absolute value matrix containing the absolute values ​​of negative elements. In this way, the information represented by the negative elements in the Q and K matrices can be preserved during the approximation of the softmax function using the kernel function, thereby improving the accuracy of the linear self-attention calculation results and ultimately improving the accuracy of the self-attention output.

[0163] In this embodiment, when calculating self-attention between elements within each segment, a dual-positional encoding operator can be used to positionally encode the Q-matrix and K-matrix of each segment based on the position of each element within the input sequence. Each vector in the encoded Q-matrix and K-matrix will contain the absolute position information of the corresponding element. Based on this, when calculating self-attention using the encoded Q-matrix and K-matrix, the correlation between elements can be captured more accurately; that is, a more precise correlation or similarity between elements can be obtained, thereby improving the accuracy of the self-attention calculation results.

[0164] Finally, it should be noted that the embodiments of this application mainly describe the process of calculating self-attention using a linear self-attention algorithm after segmenting the input sequence. In some possible cases, the linear self-attention algorithm provided in the embodiments of this application can also be used to calculate self-attention for unsegmented input sequences. That is, for unsegmented input sequences, the positive and negative elements in the Q and K matrices of the input sequence can be split into two matrices, and the absolute values ​​of the elements of the negative matrix containing negative elements can be taken. Then, a kernel function is used to process the positive matrix containing positive elements and the absolute value matrix containing the absolute values ​​of negative elements, thereby improving the accuracy of linear self-attention calculation. The relevant implementation process can be referred to in the descriptions of S4021 to S4023 above, and will not be repeated here.

[0165] The sequence processing apparatus provided in the embodiments of this application will be described next.

[0166] Figure 8 This is a schematic diagram of a sequence processing device provided in an embodiment of this application. This sequence processing device can be deployed in the aforementioned... Figure 1 The computing device shown or Figure 2 In the cluster of computing devices shown. For example... Figure 8 As shown, the sequence processing device 800 includes: a segmentation module 801, an intra-segment self-attention calculation module 802, an inter-segment self-attention calculation module 803, and a self-attention fusion module 804.

[0167] The segmentation module 801 is used to execute S401 in the aforementioned embodiment; the intra-segment self-attention calculation module 802 is used to execute S402 in the aforementioned embodiment; the inter-segment self-attention calculation module 803 is used to execute S403 and S404 in the aforementioned embodiment; and the self-attention fusion module 804 is used to execute S405 in the aforementioned embodiment.

[0168] Optionally, the input sequence represents the data to be processed as text, where the data unit is a word in the text and the multiple elements are feature vectors of multiple words in the text; or, the data to be processed is an image, where the data unit is an image region in the image and the multiple elements are feature vectors of multiple image regions in the image; or, the data to be processed is video, where the data unit is a video unit in the video and the multiple elements are feature vectors of multiple video units in the video; or, the data to be processed is speech data, where the data unit is a speech segment in the speech data and the multiple elements are feature vectors of multiple speech segments in the speech data.

[0169] Optionally, the feature information of each segment includes a feature key vector and a feature value vector; the inter-segment self-attention calculation module 803 is specifically used to: extract features from the key matrix of the first segment among multiple segments to obtain the feature key vector of the first segment, wherein the key matrix of the first segment includes the key vectors of the elements in the first segment; and extract features from the value matrix of the first segment to obtain the feature value vector of the first segment, wherein the value matrix of the first segment includes the value vectors of the elements in the first segment.

[0170] Optionally, the inter-segment self-attention calculation module 803 is specifically used to: perform self-attention calculation based on the query matrix of the input sequence, the feature key vectors and feature value vectors of multiple segments, to obtain the inter-segment self-attention corresponding to each element in the input sequence, wherein the query matrix of the input sequence includes the query vectors of the elements in the input sequence.

[0171] Optionally, the intra-segment self-attention calculation module 803 is specifically used to: perform linear self-attention calculation based on the query matrix, key matrix and value matrix of the first segment among multiple segments, to obtain the intra-segment self-attention corresponding to each element in the first segment, wherein the query matrix of the first segment includes the query vector of the element in the first segment, the key matrix of the first segment includes the key vector of the element in the first segment, and the value matrix of the first segment includes the value vector of at least two elements in the first segment.

[0172] Optionally, the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix. The intra-segment self-attention calculation module 802 is further configured to: based on the position of each element in the first segment in the input sequence, use a first positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively to obtain the first query matrix and the first key matrix; based on the position of each element in the first segment in the input sequence, use a second positional encoding operator to perform positional encoding on the original query matrix and the original key matrix of the first segment respectively to obtain the second query matrix and the second key matrix.

[0173] Optionally, the intra-segment self-attention calculation module 802 is specifically used to: obtain the positive and negative matrices corresponding to the query matrix of the first segment and the positive and negative matrices corresponding to the key matrix of the first segment, wherein the positive matrix includes the positive elements in the corresponding matrix and the negative matrix includes the negative elements in the corresponding matrix; determine the absolute value matrix of the negative matrix corresponding to the query matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the key matrix of the first segment, wherein the absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix; and perform linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix.

[0174] Optionally, when the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix, the intra-segment self-attention calculation module 802 is specifically used to: calculate the first self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the first query matrix, the absolute value matrices of the positive and negative matrices corresponding to the first key matrix, and the value matrices of the positive and negative matrices corresponding to the first key matrix; calculate the second self-attention of each element in the first segment based on the absolute value matrices of the positive and negative matrices corresponding to the second query matrix, the absolute value matrices of the positive and negative matrices corresponding to the second key matrix; and determine the intra-segment self-attention of each element in the first segment based on the first self-attention and the second self-attention of each element in the first segment.

[0175] In this embodiment, for an input sequence comprising multiple segments, self-attention between elements within each segment can be calculated to obtain intra-segment self-attention containing local context information. Furthermore, feature information from multiple segments can be extracted and used to calculate self-attention between each element and multiple segments, resulting in inter-segment self-attention containing global context information. Based on this, the self-attention output obtained from intra-segment and inter-segment self-attention of multiple elements will simultaneously contain the local context information of each element within the segment and the global context information outside the segment where each element resides. Therefore, the sequence processing method provided in this embodiment improves the accuracy of the self-attention output while minimizing the computational and memory overhead of self-attention calculation, thereby improving the accuracy of the output sequence predicted based on the self-attention output.

[0176] It should be noted that the module division in the sequence processing apparatus provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0177] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a router, switch, or other network device) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] Furthermore, the sequence processing apparatus and sequence processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0179] It is worth noting that, in some embodiments, the sequence processing device 800 described above can be deployed in the aforementioned... Figure 1In the computing device 100 shown, for example, the memory 102 of the computing device 100 may store executable program code for implementing the functions of each module of the sequence processing apparatus 800 described above. The processor 101 can implement the sequence processing method provided in the embodiments of this application by executing the executable program code.

[0180] For example, in one possible application scenario, the computing device 100 can deploy an AI model based on a transformer architecture. This AI model can be an encoder-only model, a decoder-only model, or an encoder-decoder model. Furthermore, this AI model can be a pre-trained model or a model yet to be trained. Based on this, any one or more self-attention layers in the encoder and / or decoder of this AI model can be implemented using the sequence processing device 800 described above.

[0181] In other embodiments, the sequence processing device 800 described above may be deployed in the aforementioned... Figure 2 In the cluster of computing devices shown.

[0182] In one possible implementation, each computing device in the computing device cluster may be equipped with a sequence processing device 800. For example, each computing device may have executable program code stored in its memory to implement the functions of various modules of the sequence processing device 800. In this case, the processors of multiple computing devices may execute the executable program code in their own memories to implement the sequence processing method described above.

[0183] For example, in one possible application scenario, a computing device cluster comprises multiple computing devices, each deploying the same AI model to be trained. Each computing device can train its own AI model using different training samples. This AI model can be an encoder-only model, a decoder-only model, or an encoder-decoder model. In this case, one or more self-attention layers in the AI ​​models of each computing device can be implemented using the sequence processing device 800 described above.

[0184] In another possible implementation, the memories of different computing devices in the computing device cluster can store different executable program codes, each used to implement a portion of the functionality of the sequence processing device. In this case, the various computing devices in the computing device cluster can communicate with each other via a network.

[0185] For example, a computing device cluster includes three computing devices. The executable program code stored in the memory of the first computing device implements the functions of a segmentation module 801 and a self-attention fusion module 804. The executable code stored in the memory of the second computing device implements the function of an intra-segment self-attention calculation module 802. The executable code stored in the memory of the third computing device implements the function of an inter-segment self-attention calculation module 803. Based on this, the first computing device can execute the program code implementing the function of the segmentation module 801 to divide the input sequence into multiple segments and send these segments to the second and third computing devices. The second computing device can execute the program code implementing the function of the intra-segment self-attention calculation module 802 to obtain the intra-segment self-attention calculation result based on the multiple segments and return the result to the first computing device. The third computing device can execute the program code implementing the function of the inter-segment self-attention calculation module 803 to obtain the inter-segment self-attention calculation result based on the multiple segments and return the result to the first computing device. The first computing device can fuse the intra-segment self-attention calculation results and inter-segment self-attention calculation results by executing program code that implements the function of the self-attention fusion module, thereby obtaining the self-attention output result.

[0186] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0187] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for ease of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0188] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0189] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sequence processing method, characterized in that, The method includes: The input sequence is divided into multiple segments, each segment including at least two elements. The multiple elements in the input sequence are used to represent multiple data units in the data to be processed, and the multiple elements correspond one-to-one with the multiple data units. Calculate the self-attention between elements within each segment to obtain the segment self-attention corresponding to each element; Extract feature information from the multiple segments, and calculate the self-attention between each element in the input sequence and the multiple segments based on the feature information of the multiple segments, to obtain the inter-segment self-attention corresponding to each element in the input sequence; The self-attention output result is determined based on the intra-segment self-attention and inter-segment self-attention corresponding to multiple elements in the input sequence.

2. The method according to claim 1, characterized in that, The data to be processed is text, the data unit is a word in the text, and the multiple elements are feature vectors of multiple words in the text; or, the data to be processed is an image, the data unit is an image region in the image, and the multiple elements are feature vectors of multiple image regions in the image; or, the data to be processed is video, the data unit is a video unit in the video, and the multiple elements are feature vectors of multiple video units in the video; or, the data to be processed is speech data, the data unit is a speech segment in the speech data, and the multiple elements are feature vectors of multiple speech segments in the speech data.

3. The method according to claim 1 or 2, characterized in that, The feature information includes feature key vectors and feature value vectors; the extraction of feature information from the multiple segments includes: Feature extraction is performed on the key matrix of the first segment among the plurality of segments to obtain the feature key vector of the first segment. The key matrix of the first segment includes the key vectors of the elements within the first segment. Feature extraction is performed on the value matrix of the first segment to obtain the feature value vector of the first segment. The value matrix of the first segment includes the value vector of the elements within the first segment.

4. The method according to claim 3, characterized in that, The step of calculating the self-attention between each element in the input sequence and the multiple segments based on the feature information of the multiple segments, to obtain the inter-segment self-attention corresponding to each element in the input sequence, includes: Based on the query matrix of the input sequence, the feature key vectors and feature value vectors of the multiple segments, self-attention is calculated to obtain the inter-segment self-attention corresponding to each element in the input sequence. The query matrix of the input sequence includes the query vectors of the elements in the input sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The calculation of self-attention between elements within each segment to obtain the intra-segment self-attention corresponding to each element includes: Based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments, linear self-attention is calculated to obtain the intra-segment self-attention corresponding to each element in the first segment. The query matrix of the first segment includes the query vector of the element in the first segment, the key matrix of the first segment includes the key vector of the element in the first segment, and the value matrix of the first segment includes the value vector of at least two elements in the first segment.

6. The method according to claim 5, characterized in that, The query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix. The method further includes: Based on the position of each element in the first segment in the input sequence, the original query matrix and the original key matrix of the first segment are positionally encoded using the first positional encoding operator to obtain the first query matrix and the first key matrix. Based on the position of each element in the first segment in the input sequence, the original query matrix and the original key matrix of the first segment are positionally encoded using the second positional encoding operator to obtain the second query matrix and the second key matrix.

7. The method according to claim 5 or 6, characterized in that, The linear self-attention calculation based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments includes: Obtain the positive and negative matrices corresponding to the query matrix of the first segment and the positive and negative matrices corresponding to the key matrix of the first segment, wherein the positive matrix includes positive elements in the corresponding matrix and the negative matrix includes negative elements in the corresponding matrix; Determine the absolute value matrix of the negative matrix corresponding to the query matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the key matrix of the first segment, wherein the absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix; Linear self-attention calculation is performed based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix.

8. The method according to claim 7, characterized in that, When the query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix, the linear self-attention calculation based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix includes: Based on the absolute value matrices of the positive and negative matrices corresponding to the first query matrix, the absolute value matrices of the positive and negative matrices corresponding to the first key matrix, and the value matrix, calculate the first self-attention of each element in the first segment; Based on the absolute value matrices of the positive and negative matrices corresponding to the second query matrix, the absolute value matrices of the positive and negative matrices corresponding to the second key matrix, and the value matrix, calculate the second self-attention of each element in the first segment; Based on the first self-attention and the second self-attention of each element within the first segment, the intra-segment self-attention of each element within the first segment is determined.

9. A sequence processing apparatus, characterized in that, The device includes: The segmentation module is used to divide the input sequence into multiple segments. The input sequence includes multiple elements, and each segment includes at least two elements. The multiple elements are used to represent multiple data units in the data to be processed, and the multiple elements correspond one-to-one with the multiple data units. The intra-segment self-attention calculation module is used to calculate the self-attention between at least two elements within each segment, and to obtain the intra-segment self-attention corresponding to each element. The inter-segment self-attention calculation module is used to extract feature information of the multiple segments, and calculate the self-attention between each element in the input sequence and the multiple segments based on the feature information of the multiple segments, so as to obtain the inter-segment self-attention corresponding to each element in the input sequence; The self-attention fusion module is used to determine the self-attention output result based on the intra-segment self-attention and inter-segment self-attention corresponding to multiple elements in the input sequence.

10. The apparatus according to claim 9, characterized in that, The data to be processed is text, the data unit is a word in the text, and the multiple elements are feature vectors of multiple words in the text; or, the data to be processed is an image, the data unit is an image region in the image, and the multiple elements are feature vectors of multiple image regions in the image; or, the data to be processed is video, the data unit is a video unit in the video, and the multiple elements are feature vectors of multiple video units in the video; or, the data to be processed is speech data, the data unit is a speech segment in the speech data, and the multiple elements are feature vectors of multiple speech segments in the speech data.

11. The apparatus according to claim 9 or 10, characterized in that, The feature information includes feature key vectors and feature value vectors; the inter-segment self-attention calculation module is specifically used for: Feature extraction is performed on the key matrix of the first segment among the plurality of segments to obtain the feature key vector of the first segment. The key matrix of the first segment includes the key vectors of the elements within the first segment. Feature extraction is performed on the value matrix of the first segment to obtain the feature value vector of the first segment. The value matrix of the first segment includes the value vector of the elements within the first segment.

12. The apparatus according to claim 11, characterized in that, The inter-segment self-attention calculation module is specifically used for: Based on the query matrix of the input sequence, the feature key vectors and feature value vectors of the multiple segments, self-attention is calculated to obtain the inter-segment self-attention corresponding to each element in the input sequence. The query matrix of the input sequence includes the query vectors of the elements in the input sequence.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The intra-segment self-attention calculation module is specifically used for: Based on the query matrix, key matrix, and value matrix of the first segment among the plurality of segments, linear self-attention is calculated to obtain the intra-segment self-attention corresponding to each element in the first segment. The query matrix of the first segment includes the query vector of the element in the first segment, the key matrix of the first segment includes the key vector of the element in the first segment, and the value matrix of the first segment includes the value vector of at least two elements in the first segment.

14. The apparatus according to claim 13, characterized in that, The query matrix of the first segment includes a first query matrix and a second query matrix, and the key matrix of the first segment includes a first key matrix and a second key matrix. The intra-segment self-attention calculation module is further used for: Based on the position of each element in the first segment in the input sequence, the original query matrix and the original key matrix of the first segment are positionally encoded using the first positional encoding operator to obtain the first query matrix and the first key matrix. Based on the position of each element in the first segment in the input sequence, the original query matrix and the original key matrix of the first segment are positionally encoded using the second positional encoding operator to obtain the second query matrix and the second key matrix.

15. The apparatus according to claim 13 or 14, characterized in that, The intra-segment self-attention calculation module is specifically used for: Obtain the positive and negative matrices corresponding to the query matrix of the first segment and the positive and negative matrices corresponding to the key matrix of the first segment, wherein the positive matrix includes positive elements in the corresponding matrix and the negative matrix includes negative elements in the corresponding matrix; Determine the absolute value matrix of the negative matrix corresponding to the query matrix of the first segment and the absolute value matrix of the negative matrix corresponding to the key matrix of the first segment, wherein the absolute value matrix is ​​obtained by taking the absolute value of the negative elements in the negative matrix; Linear self-attention calculation is performed based on the absolute value matrices of the positive and negative matrices corresponding to the query matrix of the first segment, the absolute value matrices of the positive and negative matrices corresponding to the key matrix of the first segment, and the value matrix.

16. A computing device, characterized in that, The computing device includes a processor for executing a computer program stored in a memory to implement the sequence processing method according to any one of claims 1 to 8.

17. A computing device cluster, characterized in that, The computing device cluster includes multiple computing devices, each computing device including a processor and a memory; the processor of the multiple computing devices is used to execute instructions stored in the memory of the multiple computing devices, so that the computing device cluster performs the sequence processing method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the sequence processing method as described in any one of claims 1 to 8.

Images