Data processing method, apparatus, computing device, and program product

Abstract

The application discloses a data processing method and device, a computing device and a program product, and belongs to the technical field of AI. In the application, for each attention head in a transformation layer in a model, a sparse mode of the attention head is determined, and then the first sequence is subjected to attention sparse processing based on the sparse mode of the attention head. The sparse mode is determined according to an attention head mode of the attention head, and the attention head mode is used to indicate the attention mode of the attention head to the input sequence, that is, to indicate which dependency relationship in the input sequence is paid attention to by the attention head. As can be seen, in the application, for different attention heads in the model, the corresponding sparse mode is used to perform attention sparse processing based on the attention mode of the attention head to the input sequence, so that the calculation cost can be reduced and the calculation time can be shortened while the calculation accuracy is guaranteed as much as possible.

Description

[0001] This application is a divisional application of Chinese patent application filed on April 29, 2025, with application number 202510561332.5 and entitled "Data Processing Method, Apparatus, Computing Device and Program Product". Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and more particularly to a data processing method, apparatus, computing device, and program product. Background Technology

[0003] In recent years, the rapid development of AI technology has driven breakthroughs in generative models, especially the combination of diffusion models and transformer architectures, which has led to significant progress in video generation technology. In video generation tasks, the computational cost of attention increases exponentially with the length of the input sequence. This high computational overhead often limits current video generation models to outputting only low-resolution, short clips, severely restricting practical applications. Therefore, optimizing the computational efficiency of attention has become crucial to overcoming the bottleneck in video generation technology. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computing device, and program product to alleviate the problems of high computational overhead and long latency in attention calculations, thereby improving the efficiency of attention calculations.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, a data processing method is provided, the method comprising: obtaining a sparse pattern of at least one attention head in a first transformation layer of a target model, the sparse pattern being determined based on the attention head pattern of the attention head and used to indicate the manner of performing attention sparse processing through the attention head; and performing attention sparse processing on a first sequence through the first transformation layer based on the sparse pattern of the at least one attention head.

[0006] The first sequence can be a sequence corresponding to multiple video frames included in a video. For example, a video frame in a video is represented by a sub-sequence, and the first sequence can include multiple sub-sequences for representing multiple video frames in the video. Alternatively, the first sequence can be a sequence corresponding to multiple image regions included in an image. For example, an image region in an image is represented by a sub-sequence, and the first sequence can include multiple sub-sequences for representing multiple image regions in the image.

[0007] In this application, for any attention head in a transform layer of the model, a sparse mode of the attention head is determined, and then attention sparse processing is performed on the first input sequence based on the sparse mode of the attention head. The sparse mode is determined according to the attention head mode of the attention head, which indicates how the attention head focuses on the input sequence; that is, it indicates which dependencies in the input sequence the attention head focuses on. Therefore, in this application, for different attention heads in the model, a corresponding sparse mode is used for attention sparse processing based on how the attention head focuses on the input sequence. This can reduce computational overhead and shorten computation time while maintaining computational accuracy as much as possible, providing support for generating higher quality, longer duration videos or higher quality images.

[0008] Optionally, in this application, the attention head mode includes a spatial head mode, a temporal head mode, and a texture head mode.

[0009] The spatial head mode refers to a mode in which the attention head focuses on the dependencies between multiple subsequences in the input sequence and the dependencies between elements within each subsequence. For example, if the sequence to be processed, such as the first sequence, includes multiple subsequences corresponding to multiple video frames, the spatial head mode attention head can capture the dependencies between multiple video frames and the dependencies between image regions within each video frame.

[0010] The temporal head mode refers to a mode in which the attention head focuses on the relationship between sequence segments at the same position in multiple subsequences of an input sequence. For example, when the sequence to be processed, such as the first sequence, includes multiple subsequences corresponding to multiple video frames, the attention head in the temporal head mode can capture the dependency relationship between image regions at the same position in multiple video frames.

[0011] The texture head mode refers to a mode other than the spatial head mode and the temporal head mode. Alternatively, the texture head mode refers to a mode used to capture irregular dependencies in a sequence.

[0012] Optionally, the process of obtaining the sparse mode of at least one attention head in the first transform layer of the target model may include: obtaining multiple sparse modes, including the spatial head sparse mode corresponding to the spatial head mode, the temporal head sparse mode corresponding to the temporal head mode, and the texture head sparse mode corresponding to the texture head mode; calculating the first attention recall rate of the target sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode using the first attention head among the at least one attention head, wherein the attention recall rate of the target sequence in any sparse mode is used to indicate the accuracy loss of the sparse attention of the target sequence in the corresponding sparse mode compared to the full attention of the target sequence; and determining the sparse mode of the first attention head based on the first attention recall rate and the second attention recall rate.

[0013] In this application, the attention recall rate corresponding to the sparse pattern can be used to characterize the accuracy loss of the attention head in the corresponding sparse pattern compared to full attention. Based on this, this application determines the degree of accuracy loss of the spatial head sparse pattern and the temporal head sparse pattern by calculating the attention recall rate of the spatial head sparse pattern and the temporal head sparse pattern. Then, based on the magnitude of the accuracy loss in these two sparse patterns, it determines which sparse pattern the attention head samples. In this way, while reducing computational overhead, the accuracy loss caused by attention sparsity processing is minimized as much as possible.

[0014] Optionally, the target sequence is an offline calibration sequence, which is different from the first sequence. Based on this, the process of calculating the first attention recall rate of the target sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode using the first attention head among the at least one attention head may include: calculating the full attention score matrix of the offline calibration sequence, the first sparse attention score matrix in the spatial head sparse mode, and the second sparse attention score matrix in the temporal head sparse mode using the first attention head; calculating the first attention recall rate corresponding to the spatial head sparse mode based on the full attention score matrix of the offline calibration sequence and the first sparse attention score matrix; and calculating the second attention recall rate corresponding to the temporal head sparse mode based on the full attention score matrix of the offline calibration sequence and the second sparse attention score matrix.

[0015] Optionally, after obtaining the sparse pattern of at least one attention head in the first transformation layer of the target model, the method further includes: storing the sparse pattern of the at least one attention head; correspondingly, the implementation process of performing attention sparse processing on the first sequence to be processed through the first transformation layer based on the sparse pattern of the at least one attention head may include: at the t-th time step, reading the stored sparse pattern of the at least one attention head, and performing attention sparse processing on the first sequence through the first transformation layer based on the read sparse pattern of the at least one attention head.

[0016] In this application, the sparse patterns of each attention head in the target model can be determined offline using an offline calibration sequence. Based on this, during online inference using the target model, the pre-calibrated sparse patterns of each attention head can be directly applied for sparsity processing, which is quick and convenient.

[0017] Optionally, the target sequence includes a first sampling sequence and a second sampling sequence corresponding to the first sequence, wherein the first sampling sequence is obtained by sampling the first sequence at intervals, and the second sampling sequence is a sequence segment in the first sequence; the process of calculating the first attention recall rate of the target sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode through the first attention head of at least one attention head may include: at the t-th time step, calculating the first attention recall rate of the first sampling sequence in the spatial head sparse mode through the first attention head, and calculating the second attention recall rate of the second sampling sequence in the temporal head sparse mode.

[0018] In this application, during the online inference process of the target model, the attention recall rate of the attention head under different sparse patterns can be calculated using the sampled sequence obtained by sampling the first sequence to be processed. Then, the sparse pattern of the attention head is determined based on the attention recall rate. This reduces the computational burden of online sparse pattern determination and makes online deployment easier. Furthermore, since the sparse pattern of the attention head is determined using the sampled sequence of the first sequence to be processed, the determined sparse pattern is more suitable for the current first sequence to be processed, resulting in better adaptability and stronger robustness.

[0019] Optionally, the method further includes: caching the sparse pattern of at least one attention head of the first transform layer determined at time step t; at time step t+1, obtaining the cached sparse pattern of at least one attention head of the first transform layer, and performing attention sparse processing on the input sequence through the first transform layer based on the sparse pattern of the at least one attention head.

[0020] In video or image generation tasks, since the data processed at adjacent time steps during model inference is similar, this application, after determining the sparse pattern of each attention head online at time step t, can also cache the determined sparse patterns of each attention head. Thus, in time step t+1, the sparse patterns of each attention head can be directly reused to perform sparse processing. This eliminates the need for online identification of the sparse patterns of each attention head, reducing computational overhead.

[0021] Optionally, the method of determining the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate may include: if at least one of the first attention recall rate and the second attention recall rate is greater than the recall rate threshold, then the sparse pattern corresponding to the maximum recall rate among the at least one attention recall rate is determined as the sparse pattern of the first attention head.

[0022] In this application, if at least one of the first and second attention recall rates is greater than a recall threshold, it indicates that the precision of at least one sparse pattern corresponding to that at least one attention recall rate meets the requirements, meaning that the precision loss of that at least one sparse pattern is relatively small. In this case, the sparse pattern with the smallest precision loss can be selected from the at least one sparse pattern as the sparse pattern of the first attention head. This allows for sparse processing of attention with minimal precision loss, reducing computational overhead.

[0023] Optionally, if both the first attention recall rate and the second attention recall rate are not greater than the recall rate threshold, then the sparse mode of the first attention head is determined to be the texture head sparse mode.

[0024] Optionally, the texture head sparse mode includes a first sparse mode. The first sparse mode is used to instruct the corresponding attention head to sample the key matrix and value matrix of the input sequence according to preset sampling parameters to obtain a sparse key matrix and a sparse value matrix, and to calculate the attention of the input sequence based on the sparse key matrix and the sparse value matrix.

[0025] In this application, under the first sparse mode, the key matrix and value matrix of the input sequence can be sampled at intervals. When the input sequence includes multiple subsequences corresponding to multiple video frames or multiple subsequences corresponding to multiple image regions in an image, each row vector in the key matrix and value matrix represents a region in the video frame or image. Therefore, sampling the key matrix and value matrix at intervals is equivalent to sampling the video frame or image at intervals. Furthermore, adjacent image regions in a video frame or image often have high similarity. Therefore, by sampling the key matrix and value matrix at intervals, compression of the key matrix and value matrix is ​​achieved with minimal loss of precision, thereby reducing computational overhead.

[0026] Optionally, the texture head sparse mode further includes a second sparse mode. The second sparse mode is used to instruct the corresponding attention head to calculate the corresponding partial attention based on a portion of the query vectors in the query matrix of the input sequence in the current time step, and to obtain the partial attention corresponding to a portion of the query vectors in the query matrix of the input sequence calculated in the target time step before the current time step, and to determine the attention of the input sequence in the current time step based on the partial attention calculated by itself and the partial attention obtained in the target time step.

[0027] The difference in the number of steps between the current time step and the target time step is no greater than the preset number of reuses.

[0028] In this application, under the second sparse mode, the attention head calculates the attention corresponding to a portion of the query vectors in the current time step. For the attention corresponding to the remaining query vectors, the calculation results of previous time steps can be reused, which can reduce computational overhead.

[0029] Optionally, the process of determining the sparse mode of the first attention head as the sparse mode of the texture head may include: if the current time step is the i-th time step, then the sparse mode of the first attention head is determined to be the second sparse mode, where i is greater than a first value and less than a second value, the first value is greater than or equal to 1, and the second value is less than or equal to the total number of time steps required for inference of the target model; if the current time step is the j-th time step, then the sparse mode of the first attention head is determined to be the first sparse mode, where j is not greater than the first value or not less than the second value.

[0030] In this application, in video or image generation tasks, the similarity of data processed in intermediate time steps is often high. Therefore, this second sparsity mode can be applied to the attention sparse processing in intermediate time steps, thereby reducing the computational cost of attention computation with minimal loss of precision.

[0031] Optionally, the difference in sparsity between any two sparse modes in the plurality of sparse modes is less than a sparsity difference threshold. The sparsity of a sparse mode can be used to characterize the proportion of computational reduction under that sparse mode, that is, it can also be used to characterize the computational efficiency gain under that sparse mode, or in other words, the acceleration proportion under that sparse mode. Based on this, when the difference in sparsity between any two sparse modes in the plurality of sparse modes is less than the sparsity difference threshold, the sparsity of the plurality of sparse modes is close to or the same. Thus, when attention computation of multiple attention heads in a transformer layer is implemented using multiple computing units such as multiple GPUs, although the sparse modes of different attention heads are different, because the sparsity of each sparse mode is close to or the same, the acceleration proportion (i.e., the computational efficiency gain) of each computing unit is basically close to or the same. This avoids the speed-lag problem caused by different acceleration proportions of different computing units.

[0032] Optionally, the process of obtaining multiple sparse patterns may include: obtaining multiple sparse pattern groups, each sparse pattern group including the multiple sparse patterns, wherein the sparsity difference between any two sparse patterns in a sparse pattern group is less than a sparsity difference threshold, and the sparsity difference between any two sparse patterns in different sparse pattern groups is not less than the sparsity difference threshold.

[0033] In this application, multiple sparse pattern groups can also be configured. The sparsity of sparse patterns in different sparse pattern groups varies significantly, while the sparsity of sparse patterns in the same sparse pattern group is similar. Based on this, the sparse pattern of the first attention head can be determined by simultaneously combining the attention recall and sparsity of different sparse patterns. That is, a suitable sparse pattern can be determined by combining the accuracy loss and computational efficiency gain of different sparse patterns, thereby achieving a balance between the efficiency and accuracy of attention computation.

[0034] Optionally, the process of determining the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate may include: if at least one of the first attention recall rates corresponding to the multiple spatial head sparse patterns in the multiple sparse pattern groups and the second attention recall rates corresponding to the multiple temporal head sparse patterns has an attention recall rate greater than a recall rate threshold, then the sparse pattern with the largest sparsity among the at least one sparse patterns corresponding to the at least one attention recall rate is determined as the sparse pattern of the first attention head.

[0035] In this application, when multiple sparse pattern groups are configured, at least one sparse pattern that meets the accuracy loss requirement is first searched based on the attention recall rates of the spatial head sparse patterns and temporal head sparse patterns in each sparse pattern group. If multiple sparse patterns meet the accuracy loss requirement, the sparse pattern with the highest sparsity, i.e., the sparse pattern with the greatest computational efficiency gain, can be selected. In this way, computational efficiency gains are maximized while ensuring computational accuracy.

[0036] In a second aspect, a data processing apparatus is provided, the data processing apparatus comprising at least one module, the at least one module being configured to perform the data processing method described in the first aspect above.

[0037] Thirdly, a computing device is provided, the computing device including a processor, the processor being configured to execute at least one program instruction or code stored in a memory to implement the data processing method described in the first aspect above.

[0038] Fourthly, a computing cluster is provided, the computing cluster including multiple computing devices, the multiple computing devices being used to execute the data processing method described in the first aspect above.

[0039] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computing device, cause the computing device to perform the data processing method described in the first aspect.

[0040] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computing device, causes the computing device to perform the data processing method described in the first aspect.

[0041] The technical effects achieved by the second to sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the structure of a computing cluster. Figure 3 This is a schematic diagram illustrating the dependencies between multiple video frames in a video that are focused on by an attention head in a spatial head mode, as provided in an embodiment of this application. Figure 4 This is a schematic diagram illustrating the dependency relationship between image regions at the same position in multiple video frames of a video that are focused on by the attention head of a timing head mode provided in an embodiment of this application. Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 6 A flowchart illustrating the sparse pattern of an offline calibrated attention head is provided in an embodiment of this application; Figure 7 A schematic diagram of a spatial header mask provided for an embodiment of this application; Figure 8 A schematic diagram of a timing header mask provided for an embodiment of this application; Figure 9 A flowchart illustrating another offline calibrated attention head sparse pattern provided in this application embodiment; Figure 10 This is a schematic diagram illustrating the determination of the sparse pattern of the attention head in the case of multiple sparse pattern groups, as provided in an embodiment of this application. Figure 11 A schematic diagram of a first sparse mode provided for an embodiment of this application; Figure 12 A schematic diagram of the second sparse mode provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] Before providing a detailed explanation of the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0045] In recent years, the rapid development of AI technology has driven breakthroughs in generative models, especially the combination of diffusion models and transformer architectures, which has led to significant progress in image and video generation technologies. When using such generative models to generate images or videos, the transformer modules within the model require attention calculations on the input sequence for each transformer layer.

[0046] For example, taking the diffusion transformer (DIT) model as an example, the DIT model can include a diffusion module and a transformer module. The diffusion module adds noise to the input data, resulting in noisy data. The DIT model then processes this noisy data to obtain an input sequence, which is fed into the transformer module. Each transformer layer in the transformer module performs various processing operations on the input sequence, including attention calculations, to predict whether to remove noise or clean data from the noisy data, thus outputting denoised data.

[0047] In this application, when performing attention calculations on the input sequence in each transformer layer, the computational cost and time consumption increase exponentially with the length of the input sequence. This often results in current generative models only being able to output low-resolution images or low-resolution, short-duration video clips, severely limiting their practical application in image and video generation. To address this, this application provides a data processing method. In this method, for any attention head in a transformer layer of the model, the sparsity mode of the attention head is determined, and then attention sparse processing is performed on the first input sequence based on the sparsity mode of the attention head. The sparsity mode is determined based on the attention head mode, which indicates how the attention head focuses on the input sequence, i.e., which aspect of the input sequence's dependencies the attention head focuses on. Therefore, in this application, for different attention heads in the model, a corresponding sparsity mode is used for attention sparse processing based on the attention head's focus on the input sequence. This approach can reduce computational overhead and shorten computation time while maintaining computational accuracy as much as possible.

[0048] The data processing method provided in this application can be applied to a single computing device. For example, Figure 1 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 1 As shown, the computing device may include a processor 101, a memory 102, a communication bus 103, and a communication interface 104. It should be noted that... Figure 1 The illustrated device structure does not constitute a limitation on the computing device. A computing device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Furthermore, the embodiments of this application do not limit the number of processors 101 and memory 102 included in the computing device 100. The following is in conjunction with… Figure 1 A detailed description of each component of a computing device: Processor 101 is the control and computing center of the computing device. Processor 101 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), neural network processing unit (NPU), microprocessor (MP), digital signal processor (DSP).

[0049] Memory 102 may be high bandwidth memory (HBM), read-only memory (ROM), or other types of static storage devices capable of storing static information and instructions; random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computing device, but not limited thereto. Memory 102 may exist independently and be connected to processor 101 via communication bus 103. Alternatively, memory 102 may be integrated with processor 101.

[0050] It should be noted that the memory 102 stores executable program code, and the processor 101 executes the executable program code to perform the data processing method described in the following embodiments. That is, the memory 102 stores instructions for performing the data processing method described below.

[0051] In one example, the CPU included in processor 101 can execute executable program code in memory 102 to implement the data processing method described in the following embodiments.

[0052] In another example, different parts of the executable program code in memory 102 may be executed by the CPU, GPU, and NPU included in processor 101 to collaboratively implement the data processing method in the following embodiments.

[0053] For example, the CPU can execute a portion of the program code to implement the operation of obtaining the sparse pattern of each attention head in the data processing method described below. Correspondingly, the GPU can perform attention sparse processing based on the sparse patterns of each attention head sent by the CPU.

[0054] Optionally, memory 102 can be used to store data generated during the execution of executable program code by processor 101 to implement data processing methods. For example, memory 102 can store sparse properties of each attention head of each transformer layer.

[0055] The communication bus 103 may include a path for transmitting information between the various components of the computing device 100. The communication bus 103 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0056] The communication interface 104 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0057] As one embodiment, the computing device may further include output devices and input devices. The output device communicates with the processor 101 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 101 and can receive input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0058] The data processing method provided in this application embodiment can also be executed by a computing cluster comprising multiple computing devices. The computing devices in this cluster can be servers, such as central servers, edge servers, or local servers in a local data center. In some embodiments, the computing devices can also be terminal devices such as desktop computers, laptops, or smartphones.

[0059] For example, see Figure 2 The computing cluster includes multiple computing devices ( Figure 2 The diagram shows two computing devices 201 and 202, the structure of which can be referenced. Figure 1 The computing device 100 shown. Furthermore, these multiple computing devices can be connected via a network to transmit intermediate results generated during the execution of the methods provided in the embodiments of this application. The network can be a wide area network (WAN) or a local area network (LAN), etc.

[0060] It should be noted that, in some embodiments, the memories of multiple computing devices in the computing cluster may each store partial program code or instructions for implementing the data processing methods in the following embodiments. In other words, a combination of multiple computing devices can jointly implement the data processing methods provided in the embodiments of this application. For example, the processor in computing device 201 in the computing cluster can execute the code or instructions stored in the memory to implement the operation of S51 in the following embodiments. Computing device 202 can execute the code or instructions stored in the memory to implement S52 in the following embodiments.

[0061] The following section explains some of the technical terms used in the embodiments of this application.

[0062] 1. Multi-head attention In the transformer architecture, the transformer layer can employ multiple attention heads for attention computation. Different attention heads capture different aspects of dependencies in the input sequence. For example, one attention head can capture the global dependencies of elements (tokens) in the sequence, where global dependencies refer to the influence of tokens at different positions in the sequence on that token. As another example, an attention head can capture the local dependencies of tokens in the sequence, where local dependencies refer to the influence of tokens within the same local region on that token.

[0063] 2. Attention head pattern Attention head patterns refer to the attention distribution patterns that different attention heads automatically learn during training, exhibiting specific tendencies or functions. Different attention head patterns reflect the behavioral differences among attention heads in capturing different aspects of dependencies in the input sequence. In other words, attention head patterns indicate which aspects of dependencies the attention head focuses on in the input sequence, or rather, how the attention head focuses on the input sequence.

[0064] This application provides three attention head modes: spatial head mode, temporal head mode, and textural head mode.

[0065] Spatial head mode refers to the attention head focusing on the dependencies between multiple subsequences in an input sequence and the dependencies between tokens within each subsequence. For example, in an input sequence that includes multiple subsequences corresponding to multiple video frames, the spatial head mode attention head captures the dependencies between multiple subsequences to obtain the dependencies between the multiple video frames corresponding to those subsequences. In addition, for any given subsequence, the spatial head mode attention head can also capture the dependencies between tokens within that subsequence to obtain the dependencies between the various image regions in the video frame represented by that subsequence.

[0066] For example, see Figure 3 The video includes four video frames, 31 to 34, arranged chronologically. These four frames are represented by four subsequences, which form an input sequence. The spatial head mode's attention head processes this input sequence to capture the correlations between these four video frames. For example, taking video frame 34 as an example, the further away a video frame is from video frame 34, the weaker its correlation with it. For instance, the correlation between video frames 31 and 34 is weaker than the correlation between video frames 33 and 34.

[0067] The temporal head mode refers to a mode where the attention head focuses on the relationship between sequence segments at the same position within multiple subsequences of an input sequence. For example, in an input sequence that includes multiple subsequences corresponding to multiple video frames, sequence segments at the same position within the multiple subsequences are used to represent image regions at the same position in the multiple video frames. Accordingly, the attention head in the temporal head mode is used to capture the correlation between image regions at the same position in the multiple video frames.

[0068] For example, see Figure 4The video includes four video frames 41 to 44 arranged chronologically. These four frames are represented by four sub-sequences. For image regions A, B, C, and D at the same position in the four video frames, the attention head in the time-sequence head mode can capture the correlation between these four image regions by processing the corresponding sequence segments within these four sub-sequences. For example, taking image region C in video frame 43, the correlation between the corresponding image region in a video frame farther away from video frame 43 and image region C in video frame 43 is weaker. For instance, the correlation between image region A and image region C is weaker than the correlation between image region B and image region C.

[0069] It should be noted that the dependencies captured by both the spatial header mode and the temporal header mode are regular dependencies. For example, when capturing global spatial dependencies, the farther a video frame is from the current video frame, the smaller its influence on the current video frame; that is, the weaker the correlation between the two. Similarly, when capturing the correlation between image regions at the same location in multiple video frames, the farther apart two video frames are, the weaker the correlation between image regions at the same location in those two video frames.

[0070] Texture head patterns refer to patterns of relationships other than the spatial head patterns and temporal head patterns that the attention head focuses on. In other words, texture head patterns are used to capture irregular dependencies in the input sequence. For example, in some cases, for the query matrix corresponding to almost all subsequences in the input sequence, the key matrix corresponding to certain specific subsequences always corresponds to a high attention score. Here, the attention score can also be called attention weight, which is used to indicate the correlation between subsequences. The higher the attention score, the higher the correlation between them. In other words, almost all video frames are highly correlated with a few specific video frames, or, almost all image regions at a certain position in video frames are highly correlated with image regions at corresponding positions in a few specific video frames.

[0071] 3. Sparse mode The sparsity mode is used to indicate the method of performing sparse attention processing. In this application, different attention head modes correspond to different sparsity modes based on their characteristics. The sparsity mode can be characterized by specific sparsity parameters. For example, the sparsity mode can be characterized by the bandwidth of the mask.

[0072] The data processing method provided in the embodiments of this application will be described in detail below.

[0073] Figure 5This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be executed by one or more computing devices; the following description uses the execution of the data processing method by a single computing device as an example. See also... Figure 5 The process includes the following steps: S51: Obtain the sparse pattern of at least one attention head of the first transformation layer in the target model.

[0074] In this embodiment, a target model is deployed in the computing device. This target model refers to a generative model based on a transformer architecture; for example, the target model can be a DIT model. The target model includes a transformer module, which comprises one or more transformer layers, each of which may have at least one attention head. The first transformer layer is any one of these transformer layers and has one or more attention heads.

[0075] In one possible implementation, the computing device can determine the sparse pattern of each attention head in each transformer layer of the target model offline, before performing online inference with the target model.

[0076] Specifically, taking any attention head in the first transformer layer of the target model, such as the first attention head, as an example, the computing device can use the target sequence to identify the attention head pattern of the first attention head, and then determine the sparse pattern corresponding to the attention head pattern of the first attention head as the sparse pattern of the first attention head. Here, the target sequence can be an offline calibration sequence specifically used for offline determination of the sparse pattern of attention heads. For example, see... Figure 6 The process may include steps S510 to S512, and each step will be described in detail below.

[0077] S510: Obtain multiple preset sparse patterns.

[0078] Among them, the multiple sparse modes include the spatial header sparse mode corresponding to the spatial header mode, the temporal header sparse mode corresponding to the temporal header mode, and the texture header sparse mode corresponding to the texture header mode.

[0079] In the first scenario, the computing device is pre-configured with a set of sparse modes, including spatial header sparse mode, temporal header sparse mode, and texture header sparse mode, each with corresponding sparsity parameters. Optionally, the sparsity difference between any two of these sparse modes is less than a sparsity difference threshold; in other words, the sparsity of these multiple sparse modes is close to or the same. Sparsity indicates the percentage reduction in computational cost of attention calculation under a given sparse mode compared to the computational cost of full attention calculation. Higher sparsity results in a greater reduction in computational cost for attention calculation, leading to higher computational efficiency gains but also greater loss of computational accuracy. Conversely, lower sparsity results in a smaller reduction in computational cost for attention calculation, lower computational efficiency gains, and less loss of computational accuracy.

[0080] In this embodiment, the sparsity parameters of the spatial header sparse pattern may include a first pattern identifier for identifying the spatial header sparse pattern and a corresponding spatial header mask. This spatial header mask causes each subsequence to focus on its neighboring subsequences. For example, the bandwidth of the spatial header mask is a first bandwidth. This spatial header mask can shield subsequences outside the first bandwidth range centered on the current subsequence, that is, it causes the current subsequence to focus on subsequences within the first bandwidth range centered on itself. Optionally, the spatial header mask can also cause each subsequence in the sequence to focus on key subsequences within the sequence.

[0081] For example, Figure 7 A schematic diagram of a spatial header mask in a spatial header sparse pattern is given, wherein, Figure 7 Each thick-lined square in the diagram represents the attention score between two subsequences corresponding to two video frames. For example, the second square in the first row represents the attention score between the first and second video frames. Figure 7 The small white squares are masked squares; that is, the attention scores indicated by the white squares will be masked. The small squares with filled patterns indicate the attention scores to be retained; that is, the filled pattern squares indicate the attention scope of each subsequence. Figure 7 Taking the fifth row as an example, the attention scores of the 5th video frame, the 4th video frame, itself, the 6th video frame, the 1st video frame, and the 9th video frame are preserved. That is, the 5th video frame pays attention to its neighboring 4th and 6th video frames, and also to the 1st and 9th video frames, where the 1st and 9th video frames are the keyframes corresponding to the 5th video frame. Thus, it can be seen that in this spatially sparse mode, each video frame pays attention to its neighboring video frames. In addition, each video frame also pays attention to its own corresponding keyframe in the video.

[0082] The sparsity parameters of the timing header sparse mode may include a second mode identifier for identifying the timing header sparse mode and a corresponding timing header mask. This timing header mask causes each sequence segment in each sub-sequence to focus on its adjacent sequence segments and sequence segments at the same position in other sub-sequences. For example, the bandwidth of the timing header mask may be a second bandwidth, which indicates the range of sequence segments within a video frame that a sequence segment in a video frame focuses on.

[0083] For example, Figure 8 A schematic diagram of a timing header mask in a timing header sparse mode is given, wherein, Figure 8 Each thick square in the diagram represents the attention score between subsequences corresponding to two video frames, and the black area within each square represents the attention score between image regions in the two video frames to be retained. Therefore, in this temporal header sparse mode, image regions in each video frame pay attention to image regions at the same location in other video frames.

[0084] Furthermore, in this embodiment, the texture head sparse mode may include a first sparse mode. Based on this, the sparse parameters of the texture head sparse mode may include a third mode identifier for identifying the first sparse mode and preset sampling parameters corresponding to the third mode identifier. Using the third mode identifier and the corresponding preset sampling parameters, the attention head can be instructed to sample the key matrix and value matrix of the input sequence according to the preset sampling parameters to obtain a sparse key matrix and a sparse value matrix, and then calculate the attention of the input sequence based on the sparse key matrix and the sparse value matrix.

[0085] Optionally, the texture head sparse mode may further include a second sparse mode. Based on this, the sparse parameters of the texture head sparse mode may further include a fourth mode identifier for identifying the second sparse mode. Through the fourth mode identifier, the attention head can be instructed to calculate the corresponding partial attention based on a portion of the query vectors in the query matrix of the input sequence in the current time step, and to obtain the partial attention corresponding to a portion of the query vectors in the query matrix of the input sequence calculated by the attention head in the target time step prior to the current time step. Based on the partial attention calculated by itself and the obtained partial attention in the target time step, the attention of the input sequence in the current time step is determined. Here, the partial query vectors in the query matrix of the input sequence in the current time step and the partial query vectors in the query matrix of the input sequence in the target time step are query vectors at different positions.

[0086] In this context, a time step indicates the current stage of data processing the model is in. For example, in a video generation task, the model can obtain noisy data by adding noise of a preset intensity to the input data multiple times. Based on this, the transformer module can iterate multiple times to gradually predict either the noisy or clean data. Each iteration of the transformer module can be called a time step, indicating which stage of noise prediction it is currently in.

[0087] It should be noted that, since the second sparse mode reuses the attention corresponding to a portion of the query vectors in the query matrix of the input sequence from the time steps preceding the current time step, the second sparse mode is suitable for sparse attention processing in time steps with inter-step similarity. However, during the model's inference process, inter-step similarity is often higher in intermediate time steps. Therefore, in this embodiment, the second sparse mode is suitable for sparse attention processing in intermediate time steps.

[0088] Optionally, if the texture head sparse mode includes a second sparse mode, the sparse parameters of the texture head sparse mode may further include a reuse count corresponding to the fourth mode identifier. This reuse count limits the number of steps in the target time step where the attention calculated by the attention head is allowed to be reused. That is, by using this reuse count, it can be determined whether the attention calculated in the previous target time step can be reused in the current time step. Therefore, by limiting the time step range to which attention reuse is applicable through this reuse count, the accuracy loss caused by reuse in a large number of consecutive time steps can be reduced.

[0089] For example, the reuse count is 2. Based on this, assuming the target time step is the t-th time step, the full attention of the sequence is calculated using the attention head at the t-th time step. Furthermore, at the (t+1)-th and (t+2)-th time steps, if the sparsity pattern of the attention head is the second sparse pattern, then a portion of the attention from the t-th time step can be reused. However, at the (t+3)-th time step, if the sparsity pattern of the attention head is still the second sparse pattern, then the attention calculated at the t-th time step is no longer allowed to be reused.

[0090] In the second scenario, the computing device can be pre-configured with multiple sparse pattern groups. Each sparse pattern group includes sparse patterns corresponding to various attention head patterns. The sparse patterns corresponding to the same attention head pattern are different in different sparse pattern groups, and the sparsity difference between any two sparse patterns in a sparse pattern group is less than a sparsity difference threshold. In other words, each sparse pattern group includes a set of sparse patterns with similar or identical sparsity corresponding to various attention head patterns.

[0091] For example, each sparse mode group may include a spatial head sparse mode, a temporal head sparse mode, and a texture head sparse mode. The sparsity of the spatial head sparse mode, temporal head sparse mode, and texture head sparse mode within each sparse mode group is close to or the same. The sparsity of the sparse modes in different sparse mode groups may differ or vary significantly. Each sparse mode within each sparse mode group has sparse parameters. The sparse parameters of the sparse modes corresponding to the same attention head mode in different sparse mode groups are different. For example, the sparse parameters of the spatial head sparse modes in different sparse mode groups are different. The implementation of the sparse parameters of each sparse mode can be found in the previous section and will not be repeated here.

[0092] S511: Using the first attention head, calculate the first attention recall rate of the offline calibrated sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode.

[0093] The attention recall rate corresponding to any sparse pattern can be used to characterize the degree of precision loss of sparse attention obtained under that sparse pattern. The higher the attention recall rate, the smaller the precision loss of sparse attention under that sparse pattern compared to full attention. The lower the attention recall rate, the greater the precision loss of sparse attention under that sparse pattern compared to full attention.

[0094] It should be noted that the attention of each token in the sequence depends on the attention score between that token and other tokens. Sparse attention is calculated based on the attention score between each token and some tokens in the sequence, while full attention is calculated based on the attention score between each token and all tokens in the sequence. Therefore, the difference in accuracy between sparse attention and full attention can be characterized by comparing the sparse attention score and the full attention score. Based on this, in this embodiment, the computing device calculates the full attention score matrix of the offline calibration sequence, the first sparse attention score matrix in spatial head sparse mode, and the second sparse attention score matrix in temporal head sparse mode using a first attention head. A first attention recall rate is calculated based on the first sparse attention score matrix and the full attention score matrix, and a second attention recall rate is calculated based on the second sparse attention score matrix and the full attention score matrix.

[0095] For example, the computing device can calculate the query matrix and key matrix of the offline calibration sequence using a first attention head, and calculate the full attention score matrix of the offline calibration sequence based on the query matrix and key matrix of the offline calibration sequence.

[0096] When a set of sparse patterns is configured in the computing device, the device can calculate a first sparse attention score matrix based on the query matrix and key matrix of the offline calibration sequence, and the spatial header mask included in the sparse parameters of the spatial header sparse pattern in the set of sparse patterns. Then, based on the query matrix and key matrix of the offline calibration sequence, and the temporal header mask included in the sparse parameters of the temporal header sparse pattern in the set of sparse patterns, the device can calculate a second sparse attention score matrix. Subsequently, the computing device can determine the first attention recall corresponding to the spatial header sparse pattern based on the first sparse attention score matrix and the full attention score matrix, and determine the second attention recall corresponding to the temporal header sparse pattern based on the second sparse attention score matrix and the full attention score matrix.

[0097] It should be noted that the first sparse attention score matrix includes the attention score between each token in the offline calibration sequence and a subset of tokens in the same sequence, i.e., the attention score of the portion not masked by the spatial header mask. The full attention score matrix includes the attention score between each token and each token in the offline calibration sequence. Based on this, the computing device can calculate the sum of the attention scores in the first sparse attention score matrix to obtain the first sparse attention score sum. Then, the sum of the attention scores in the full attention score matrix is ​​calculated to obtain the full attention score sum. Finally, the ratio of the first sparse attention score sum to the full attention score sum is determined as the first attention recall rate. Similarly, the second attention recall rate can also be calculated using the same method.

[0098] The above is an exemplary implementation of calculating attention recall according to an embodiment of this application. Optionally, attention recall can also be calculated in other ways. For example, sparse attention can be calculated using a sparse attention score matrix and a value matrix, and full attention can be calculated using a full attention score matrix and a value matrix. Then, attention recall can be calculated using sparse attention and full attention.

[0099] When a computing device is configured with multiple sparse pattern groups, for the spatial head sparse pattern and the temporal head sparse pattern in each sparse pattern group, the computing device can refer to the method described above to calculate the sparse attention of the offline calibration sequence in the corresponding sparse pattern using the first attention head. Thus, for multiple sparse pattern groups, the computing device can calculate the first sparse attention for each spatial head sparse pattern and the second sparse attention for each temporal head sparse pattern in the multiple sparse pattern groups. Based on this, for the spatial head sparse pattern and the temporal head sparse pattern in each sparse pattern group, the computing device can refer to the method described above to calculate the first attention recall rate corresponding to each spatial head sparse pattern and the second attention recall rate corresponding to each temporal head sparse pattern in the multiple sparse pattern groups.

[0100] S512: Determine the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate.

[0101] When a set of sparse patterns is configured in a computing device, the device can determine whether the first attention recall rate corresponding to the spatial head sparse pattern and the second attention recall rate corresponding to the temporal head sparse pattern are greater than a recall threshold. If at least one of the first and second attention recall rates is greater than the recall threshold, the sparse pattern corresponding to the highest recall rate among the at least one attention recall rate is determined as the sparse pattern of the first attention head. At this point, the attention head pattern corresponding to the determined sparse pattern of the first attention head is the attention head pattern of that first attention head.

[0102] For example, if both the first and second attention recall rates are greater than the recall threshold, it indicates that the accuracy of sparse attention in both the spatial head sparse mode and the temporal head sparse mode meets the requirements. Furthermore, if the sparsity of multiple sparse modes in this group is close to or the same, it indicates that the computational efficiency gains of the spatial head sparse mode and the temporal head sparse mode are close to or the same. In this case, the computing device can choose the sparse mode with higher accuracy gain (i.e., smaller accuracy loss) as the first attention head's sparse mode. A larger attention recall rate corresponds to a smaller accuracy loss in the sparse mode; therefore, the computing device can choose the sparse mode with the highest recall rate between the first and second attention recall rates as the first attention head's sparse mode. Of course, if the sparsity of multiple sparse modes in this group differs significantly, the sparse mode can be flexibly selected based on its sparsity to achieve a balance between computational accuracy and computational efficiency.

[0103] For example, if one of the first attention recall rate and the second attention recall rate has a recall rate greater than the recall rate threshold, while the other has a recall rate less than the recall rate threshold, it means that the sparse attention in one of the spatial head sparse patterns and the temporal head sparse patterns meets the precision requirement, while the sparse attention in the other sparse pattern does not meet the precision requirement. In this case, the sparse pattern that meets the precision requirement, that is, the sparse pattern corresponding to the attention recall rate greater than the recall rate threshold, can be directly identified as the sparse pattern of the first attention head.

[0104] Optionally, if both the first attention recall rate and the second attention recall rate are not greater than the recall rate threshold, it indicates that the sparse attention in both the spatial head sparse mode and the temporal head sparse mode does not meet the accuracy requirements. In this case, the sparse mode of the first attention head can be determined as the texture head sparse mode.

[0105] In some embodiments, the texture head sparse mode includes a first sparse mode but does not include a second sparse mode.

[0106] In other embodiments, the texture head sparse mode may include a first sparse mode and a second sparse mode. The second sparse mode is suitable for attention sparse processing at intermediate time steps during model inference. Based on this, in this embodiment, when both the first attention recall rate and the second attention recall rate are not greater than a recall rate threshold, the computing device can determine whether the sparse mode of the first attention head is the first sparse mode or the second sparse mode based on the position of the current time step among multiple time steps in this offline calibration process. The total number of time steps in the offline calibration process is the same as the total number of time steps in the online inference process.

[0107] For example, if the current time step is the i-th time step in this offline calibration process, then the sparse mode of the first attention head is determined to be the second sparse mode, where i is greater than a first value and less than a second value, the first value is greater than or equal to 1, and the second value is less than or equal to the total number of time steps required for inference of the target model. If the current time step is the j-th time step in this offline calibration process, then the sparse mode of the first attention head is determined to be the first sparse mode, where j is not greater than the first value or not less than the second value.

[0108] For example, if the total number of time steps required for the target model inference is 50, with a first value of 10 and a second value of 40, then if the current time step is one of the 11th to 39th time steps out of the 50, the sparse pattern of the first attention head is the second sparse pattern. If the current time step is one of the 1st to 10th time steps, or one of the 40th to 50th time steps, then the sparse pattern of the first attention head is the first sparse pattern.

[0109] Figure 9 An exemplary flowchart is shown, illustrating how a computing device determines the sparse pattern of a first attention head when a set of sparse patterns is configured in the computing device. Figure 9 As shown, mode S refers to the spatial header sparse mode, mode T refers to the temporal header sparse mode, mode Wa refers to the first sparse mode in the texture header sparse mode, and mode Wb refers to the second sparse mode in the texture header sparse mode. The computing device first calculates the total attention score A of the offline calibration sequence using the first attention head. G Next, the sum of sparse attention scores A for the offline calibrated sequence under pattern S is calculated. S And the sum of sparse attention scores A for offline calibrated sequences in pattern T. T Based on A S and A G The attention recall rate RS corresponding to pattern S was calculated based on A. T and A G Calculate the attention recall RT corresponding to pattern T. Determine if RS and RT are both greater than the recall threshold γ. If both RS and RT are greater than γ, compare their values. If RS is greater than RT, determine the sparse pattern of the first attention head as pattern S; if RS is less than RT, determine the sparse pattern of the first attention head as pattern T; if RS equals RT, determine the sparse pattern of the first attention head as either pattern S or pattern T. If RS is greater than γ and RT is not greater than γ, determine the sparse pattern of the first attention head as pattern S. If RS is not greater than γ and RT is greater than γ, determine the sparse pattern of the first attention head as pattern T. If neither RS ​​nor RT is greater than γ, determine if the current time step is an intermediate time step. If yes, determine the sparse pattern of the first attention head as pattern Wb; otherwise, determine the sparse pattern of the first attention head as pattern Wa.

[0110] When a computing device is configured with multiple sparse pattern groups, the computing device can determine whether there is at least one attention recall rate greater than the recall rate threshold among the attention recall rates corresponding to the spatial head sparse pattern and the temporal head sparse pattern included in the multiple sparse pattern groups. If there is at least one attention recall rate greater than the recall rate threshold, then the sparse pattern with the largest sparsity among the sparse patterns corresponding to the at least one attention recall rate is determined as the sparse pattern of the first attention head.

[0111] It should be noted that if at least one attention recall rate is greater than the recall rate threshold, it means that the sparse pattern corresponding to that at least one attention recall rate meets the accuracy requirement. Since the at least one sparse pattern corresponding to that at least one attention recall rate may belong to different sparse pattern groups with different or significantly different sparsities, if all at least one sparse pattern can meet the accuracy requirement, the computing device can select the sparse pattern with the largest sparsity (i.e., the greatest computational efficiency gain) as the sparse pattern of the first attention head, thereby achieving a balance between computational efficiency and computational accuracy.

[0112] Optionally, if the attention recall rates corresponding to the spatial head sparse mode and the temporal head sparse mode in the multiple sparse mode groups are not greater than the recall rate threshold, then the texture head sparse mode with the highest sparsity in the multiple sparse mode groups is determined as the sparse mode of the first attention head.

[0113] Optionally, if the texture head sparse mode includes a first sparse mode and a second sparse mode, the computing device can also determine whether the sparse mode of the first attention head is the first sparse mode or the second sparse mode based on the current time step.

[0114] For example, see Figure 10 The computing device is configured with three sparse mode groups. Sparse mode group 1 includes spatial header sparse mode S1, temporal header sparse mode T1, and texture header sparse mode W1; sparse mode group 2 includes spatial header sparse mode S2, temporal header sparse mode T2, and texture header sparse mode W2; and sparse mode group 3 includes spatial header sparse mode S3, temporal header sparse mode T3, and texture header sparse mode W3. The sparsity of modes S1, T1, and W1 is close to or equal to the sparsity of modes S2, T2, and W2, and the sparsity of modes S3, T3, and W3 is close to or equal to the sparsity of modes S3, T3, and W3. The sparsity of modes S1, T1, and W1 is greater than that of modes S2, T2, and W2, and the sparsity of modes S2, T2, and W2 is greater than that of modes S3, T3, and W3. Based on this, the computing device can use a first attention head to calculate the sparse attention of the calibrated sequence in the spatial head sparse mode and temporal head sparse mode of each of the above-mentioned sparse mode groups, and then calculate the attention recall corresponding to each sparse mode based on each sparse attention and the full attention of the calibrated sequence. For example... Figure 10As shown, the attention recall rate is R-S1 for mode S1, R-T1 for mode T1, R-S1 for mode S1, R-W1 for mode W1, and so on. The computing device can compare each of these attention recall rates with a recall threshold γ. R-S1, R-T2, and R-S3 are all greater than γ, meaning that modes S1, T2, and S3 all meet the accuracy requirements. Furthermore, since mode S1 has the highest sparsity among modes S1, T2, and S3, meaning it offers the greatest computational efficiency benefit, S1 can be designated as the sparse mode of the first attention head.

[0115] Optionally, in some possible cases, when multiple sparse pattern groups are configured in the computing device, after determining that there is at least one sparse pattern in the multiple sparse pattern groups with a corresponding attention recall rate greater than the recall rate threshold, the computing device may also determine the sparse pattern with the largest attention recall rate in the at least one sparse pattern as the sparse pattern of the first attention head.

[0116] For each attention head in each transformer layer of the target model, the computing device can refer to any of the methods described above to offline calibrate the sparse pattern of each attention head. Afterwards, the computing device can store the sparse pattern of each attention head in each transformer layer.

[0117] For example, a computing device can store the layer identifier of each transformer layer, the identifier of each attention head in that transformer layer, and the sparse pattern of each attention head.

[0118] In another possible implementation, the computing device can also determine the sparse pattern of each attention head in each transformer layer online during online inference through the target model.

[0119] Specifically, taking any attention head in the first transformer layer of the target model, such as the first attention head, as an example, at the t-th time step, the computing device can use the target sequence to identify the attention head pattern of the first attention head, and then determine the sparse pattern corresponding to the attention head pattern of the first attention head as the sparse pattern of the first attention head.

[0120] The target sequence includes a first sampled sequence and a second sampled sequence corresponding to the first sequence to be processed by the first transformer layer at time step t. The first and second sampled sequences are obtained by sampling the first sequence using different sampling methods. For example, the first sampled sequence can be obtained by interval sampling of the first sequence, while the second sampled sequence can be obtained by sampling local continuous segments of the first sequence.

[0121] In this embodiment, the computing device can calculate the query matrix, key matrix, and value matrix corresponding to the first sequence using a first attention focus. The query matrix includes multiple query vectors, each representing a token in the first sequence. The key matrix includes multiple key vectors, each also representing a token in the first sequence. Based on this, the computing device can sample the query matrix and key matrix using an interval sampling method to obtain a first sub-query matrix and a first sub-key matrix. For example, the computing device can start from the first row of the query matrix and key matrix and sample a row vector every m rows, where m is greater than or equal to 1. In this case, the first sub-query matrix and the first sub-key matrix represent the first sampled sequence. In other words, the first sub-query matrix and the first sub-key matrix are the query matrix and key matrix of the first sampled sequence.

[0122] Additionally, the computing device can sample N consecutive query vectors from the query matrix of the first sequence to obtain a second sub-query matrix, and sample N consecutive key vectors from the key matrix of the first sequence at the positions corresponding to the N query vectors to obtain a second sub-key matrix. For example, the first N vectors can be sampled from the query matrix and the key matrix, respectively. In this case, the second sub-query matrix and the second sub-key matrix are used to represent the second sampling sequence. In other words, the second sub-query matrix and the second sub-key matrix are the query matrix and key matrix of the second sampling sequence.

[0123] In summary, the embodiments of this application can achieve sampling of the first sequence by sampling the query matrix and key matrix of the first sequence, thereby obtaining a sampled sequence. Optionally, in some possible implementations, the computing device can also directly sample the tokens in the first sequence to obtain the corresponding sampled sequences. Then, the query matrix and key matrix of each sampled sequence are calculated using the first attention head.

[0124] For example, the computing device can acquire multiple preset sparsity patterns. A first attention recall rate of the first sampled sequence in the spatial head sparse pattern is calculated using a first attention head, and a second attention recall rate of the second sampled sequence in the temporal head sparse pattern is calculated. Then, based on the first and second attention recall rates, the sparsity pattern of the first attention head is determined.

[0125] The implementation methods for computing devices to obtain various sparse patterns can be found in the relevant introduction to S510 mentioned above, and will not be repeated here.

[0126] When a set of sparse patterns is configured in the computing device, the computing device can use the query matrix and key matrix corresponding to the first and second sampling sequences, respectively, and refer to the method described in S511 above, to calculate the first full attention score matrix corresponding to the first sampling sequence and the second full attention score matrix corresponding to the second sampling sequence using the first attention head. Based on the spatial header mask in the sparse parameters of the spatial header sparse pattern, the first sparse attention score matrix of the first sampling sequence in the spatial header sparse pattern is calculated. Based on the temporal header mask in the sparse parameters of the temporal header sparse pattern, the second sparse attention score matrix of the second sampling sequence in the temporal header sparse pattern is calculated. On this basis, the computing device calculates the first attention recall corresponding to the spatial header sparse pattern based on the first sparse attention score matrix and the first full attention score matrix, and calculates the second attention recall corresponding to the temporal header sparse pattern based on the second sparse attention score matrix and the second full attention score matrix. Then, the computing device can refer to the method described in S512 above, and determine the sparse pattern of the first attention head based on the first attention recall and the second attention recall.

[0127] When multiple sparse patterns are configured in the computing device, for both the spatial head sparse pattern and the temporal head sparse pattern in each sparse pattern group, the computing device can refer to the above method to calculate the first attention recall rate corresponding to the spatial head sparse pattern using the first sampling sequence, and calculate the second attention recall rate corresponding to the temporal head sparse pattern using the second sampling sequence. Then, the computing device can refer to the processing method described in S512 above for multiple sparse patterns, and determine the sparse pattern of the first attention head based on the attention recall rates corresponding to the spatial head sparse pattern and the temporal head sparse pattern in each sparse pattern group.

[0128] The above mainly describes the process by which the computing device determines the sparse pattern of the first attention head in the first transformer layer of the target model online at time step t during the online inference process of the target model. At time step t, for each attention head in each transformer layer of the target model, the computing device can determine the sparse pattern of each attention head using the above method.

[0129] Optionally, in video or image generation tasks, the data processed at adjacent time steps during model inference has similarity. Based on this, after determining the sparse pattern of the first attention head online, the computing device can also cache the sparse patterns of each attention head determined in the t-th time step (i.e., the current time step). In this way, in the (t+1)-th time step, the computing device can directly reuse the sparse patterns of each attention head to perform sparse processing, eliminating the need for online identification of the sparse patterns of each attention head and reducing computational overhead.

[0130] For example, at time step t, the computing device can store the transformer layer identifier, the attention head identifier, and the determined sparse pattern of the attention head in a cache. Then, at time step t+1, for any attention head in any transformer layer, the computing device can directly obtain the cached sparse pattern of the attention head based on the transformer layer identifier and the attention head identifier, and then perform attention sparse processing based on the obtained sparse pattern.

[0131] S52: Based on the sparse pattern of each attention head, perform attention sparse processing on the first sequence to be processed through the first transformation layer.

[0132] After obtaining the sparse patterns of each attention head in the first transformer layer, at time step t in the online inference process, the first transformer layer can perform attention sparse processing on the first sequence to be processed based on the sparse patterns of each attention head. The process of performing attention sparse processing with the first attention head will be illustrated below.

[0133] The first sequence can be a sequence corresponding to multiple video frames included in a video. For example, each video frame can be represented by a subsequence including multiple tokens, and the first sequence includes multiple subsequences corresponding to multiple video frames. Alternatively, the first sequence can also be a sequence corresponding to multiple image regions included in an image. For example, each image region can be represented by a subsequence including multiple tokens, and the first sequence includes multiple subsequences for representing multiple image regions.

[0134] In the first example, the sparse mode of the first attention head is the spatial head sparse mode. The computing device can calculate the query matrix, key matrix, and value matrix of the first sequence using the first attention head. Then, using the spatial head mask in the sparse parameters of the spatial head sparse mode, the query matrix and key matrix of the first sequence, the sparse attention score is calculated. Based on the sparse attention score and the value matrix, the sparse attention corresponding to the first sequence is calculated.

[0135] In the second example, the sparse mode of the first attention head is the time-series head sparse mode. The computing device can calculate the query matrix, key matrix, and value matrix of the first sequence using the first attention head. Then, using the time-series head mask, the query matrix, and the key matrix of the first sequence from the sparse parameters of the time-series head sparse mode, the sparse attention score is calculated. Based on the sparse attention score and the value matrix of the first sequence, the sparse attention corresponding to the first sequence is calculated.

[0136] In the third example, the sparse mode of the first attention head is the first sparse mode in the texture head sparse mode. The computing device calculates the query matrix, key matrix, and value matrix of the first sequence using the first attention head. Then, based on the preset sampling parameters included in the sparse parameters of the first sparse mode, the key matrix and value matrix of the first sequence are sampled respectively to obtain the sparse key matrix and sparse value matrix. Based on the query matrix, sparse key matrix, and sparse value matrix of the first sequence, the sparse attention corresponding to the first sequence is calculated.

[0137] It should be noted that, in this embodiment, the preset sampling parameter in the sparse parameters of the first sparse mode is used to indicate that the image regions represented by each row vector in the key matrix and value matrix are sampled at intervals in the image or video frame. The image regions represented by any two row vectors in the sparse key matrix and sparse value matrix may not be adjacent in the video frame. For example, the preset sampling parameter can indicate that each row vector in the key matrix and value matrix is ​​arranged in two dimensions according to the position of the corresponding image region in the image or video frame. Then, the computing device can sample the non-adjacent row vectors in the two-dimensionally arranged key matrix and value matrix to obtain the sparse key matrix and sparse value matrix. Since adjacent image regions in an image or video frame have high similarity, sampling the key matrix and value matrix at intervals according to the position of the image region corresponding to each row vector in the image or video frame can compress the key matrix and value matrix with minimal loss of precision, thereby reducing computational overhead.

[0138] For example, see Figure 11Image 1100 comprises 16 image regions. Both the key matrix K and the value matrix V contain 16 row vectors. A row vector in either the key matrix K or the value matrix V represents a small square region in image 1100. The i-th row vector in both the key matrix K and the value matrix V represents the i-th image region in image 1100, where the i-th image region is determined in a left-to-right, top-to-bottom order. For example, k1 and v1 represent the first image region in the upper left corner of image 1100, k2 and v2 represent the second image region in the first row of image 1100, and so on. Based on this, according to the position of each row vector in the key matrix K and the value matrix V corresponding to the image region in image 1100, the key matrix K and the value matrix V can be arranged in a two-dimensional manner. The two-dimensional arrangement of the key matrix K and the value matrix V is as follows: Figure 11 As shown in the diagram. Then, using a checkerboard-like staggered sampling method, the non-adjacent row vectors in the two-dimensionally arranged key matrix K and value matrix V are sampled to obtain the following... Figure 11 The sparse bond matrix K' and sparse value matrix V' shown are used to represent multiple non-adjacent black square regions arranged in a checkerboard pattern in image 1100. In other words, information about the image regions in the image is preserved in a checkerboard format in the sparse bond matrix and sparse value matrix.

[0139] In the fourth example, the sparse mode of the first attention head is the second sparse mode in the texture head sparse mode. At the current time step (i.e., the t-th time step), the computing device calculates the query matrix, key matrix, and value matrix of the first sequence using the first attention head. Then, based on a portion of the query vectors, key matrix, and value matrix in the query matrix of the first sequence at the current time step, the computing device can calculate a partial attention for the first sequence. For the remaining query vectors in the query matrix of the first sequence, the computing device can obtain the partial attention corresponding to the query vectors at the same positions as the remaining query vectors in the query matrix of the input sequence calculated by the first attention head at the target time step before the current time step. The partial attention calculated by the device itself and the partial attention obtained in the target time step are determined as the sparse attention of the first sequence. Here, the target time step refers to the time step closest to the current time step where the full attention of the sequence is calculated using the first attention head.

[0140] For example, see Figure 12 At time step t, the query matrix Qt of the first sequence includes 2N row vectors, and the computing device can obtain the first N query vectors in the query matrix Qt. to We obtain the sparse query matrix Qt'. Based on this sparse query matrix Qt', the key matrix and value matrix of the first sequence, we calculate the N attention points corresponding to the first N query vectors in the query matrix of the first sequence. to Additionally, the computing device can obtain the last N attentions from the 2N attentions calculated by the first attention head in the previous time step based on the 2N query vectors in the query matrix Qt-1. to The N attentions calculated by the current step are concatenated with the N attentions from the previous time step to obtain the sparse attention of the first sequence. Thus, in the second sparse mode, the attention corresponding to a portion of the query vectors in the query matrix is ​​calculated in the current time step, while the attention corresponding to the remaining query vectors reuses the calculation results from the previous time step.

[0141] It should be noted that in video or image generation tasks, the similarity of data processed in intermediate time steps is often high. Therefore, in this embodiment, for intermediate time steps in the inference process, based on the characteristic of high inter-step similarity, a second sparse mode can be adopted to achieve attention sparse processing by reusing the calculation results of the previous time step. In this way, the computational overhead and memory usage of attention calculation can be reduced with minimal loss of precision, thereby better supporting the generation of higher quality and longer videos.

[0142] Optionally, in some possible cases, the sparsity parameter of the second sparse mode includes the number of reuses. Based on this, after determining that the sparse mode of the first attention head is the second sparse mode at the current time step, if the step difference between the current time step and the target time step is not greater than the number of reuses, the computing device can reuse a portion of the attention calculated at the target time step using the method described above. If the difference between the current time step and the target time step is greater than the number of reuses, the computing device may not perform the sparse processing operation based on the second sparse mode, but instead calculate the full attention of the first sequence using the first attention head and store the full attention so that a portion of the full attention can be reused in subsequent time steps.

[0143] In some embodiments, the pre-configured texture head sparse mode in the computing device is a first sparse mode, meaning that the texture head sparse mode does not include a second sparse mode. Thus, when attention sparse processing is performed at the current time step based on the sparse modes of each attention head, there will be no reuse of attention calculation results from the previous time step. In other words, attention heads will not perform attention sparse processing based on inter-step similarity. Furthermore, the computing device can also superimpose cache-type sparse algorithms while performing attention sparse processing based on the sparse modes of each attention head to further improve computational efficiency.

[0144] In this embodiment, for any attention head in a transform layer of the model, a sparse mode of the attention head is determined, and then attention sparse processing is performed on the first input sequence based on the sparse mode of the attention head. The sparse mode is determined according to the attention head mode of the attention head, which indicates how the attention head focuses on the input sequence; that is, it indicates which aspect of the dependencies in the input sequence the attention head focuses on. Therefore, in this embodiment, for different attention heads in the model, a corresponding sparse mode is used for attention sparse processing based on how the attention head focuses on the input sequence. This can reduce computational overhead and shorten computation time while ensuring computational accuracy as much as possible, thereby better supporting the generation of higher quality and longer duration videos or high-quality images.

[0145] This application provides not only sparse modes corresponding to the spatial head mode and temporal head mode for capturing regular dependencies in a sequence, but also a first sparse mode corresponding to the texture head mode for capturing irregular dependencies in a sequence. In the first sparse mode, the key matrix and value matrix of the input sequence can be sampled at intervals. When the input sequence is a sequence corresponding to multiple video frames or multiple image regions in an image, each row vector in the key matrix and value matrix represents a region in the video frame or image. Therefore, sampling the key matrix and value matrix at intervals is equivalent to sampling the video frame or image at intervals. In video frames or images, adjacent image regions often have high similarity. Therefore, by sampling the key matrix and value matrix at intervals, compression of the key matrix and value matrix is ​​achieved with minimal loss of precision, thereby reducing computational overhead. In addition, this application also provides a second sparse mode corresponding to the texture head mode. In the second sparse mode, the attention head calculates the attention corresponding to a portion of the query vectors in the current time step. For the attention corresponding to the remaining query vectors, the calculation results from the previous time step can be reused, thus reducing computational overhead. Furthermore, in video or image generation tasks, the similarity of data processed in intermediate time steps is often high. Therefore, this second sparsity mode can be applied to attention sparse processing in intermediate time steps, thereby reducing the computational cost of attention computation with minimal loss of precision.

[0146] Furthermore, in this embodiment, during the online inference process of the target model, the attention recall rate of the attention head under different sparse patterns can be calculated using the sampled sequence obtained by sampling the first sequence to be processed. Then, the sparse pattern of the attention head is determined based on the attention recall rate. This reduces the computational load of online determination and makes online deployment easier. Moreover, since the sparse pattern of the attention head is determined using the sampled sequence of the first sequence to be processed, the determined sparse pattern is more suitable for the current first sequence to be processed, resulting in better adaptability and stronger robustness of the solution.

[0147] It should also be noted that the sparsity of the spatial head sparse mode, temporal head sparse mode, and texture head sparse mode provided in the embodiments of this application is close to or the same. In this way, when multiple attention heads in a transformer layer are implemented by multiple computing units such as multiple GPUs, although the sparse modes of different attention heads are different, the acceleration ratio (i.e., the gain in computing efficiency) of each computing unit is basically close to or the same because the sparsity of each sparse mode is close to or the same. In this way, the problem of different speeds and lags caused by different acceleration ratios of different computing units can be avoided.

[0148] The data processing apparatus provided in the embodiments of this application will be described next.

[0149] Figure 13 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. This data processing apparatus can be deployed in the aforementioned computing device. Figure 13 As shown, the data processing device 1300 includes a sparse pattern acquisition module 1301 and a sparse processing module 1302.

[0150] The sparse pattern acquisition module 1301 is used to execute S51 in the aforementioned embodiment, and the sparse processing module 1302 is used to execute S52 in the aforementioned embodiment.

[0151] Optionally, attention head modes include spatial head mode, temporal head mode, and texture head mode. Spatial head mode refers to the mode in which the attention head focuses on the dependencies between multiple subsequences in the input sequence and the dependencies between elements within each subsequence. Temporal head mode refers to the mode in which the attention head focuses on the relationships between sequence segments at the same position in multiple subsequences in the input sequence. Texture head mode refers to modes other than spatial head mode and temporal head mode.

[0152] Optionally, the sparse pattern acquisition module 1301 is specifically used to: acquire multiple sparse patterns, including a spatial head sparse pattern corresponding to a spatial head pattern, a temporal head sparse pattern corresponding to a temporal head pattern, and a texture head sparse pattern corresponding to a texture head pattern; calculate the first attention recall rate of the target sequence in the spatial head sparse pattern and the second attention recall rate in the temporal head sparse pattern using a first attention head among at least one attention head, wherein the attention recall rate of the target sequence in any sparse pattern is used to indicate the accuracy loss of the sparse attention of the target sequence in the corresponding sparse pattern relative to the full attention; and determine the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate.

[0153] Optionally, the target sequence is an offline calibration sequence, which is different from the first sequence. The sparse pattern acquisition module 1301 includes a sparse pattern offline calibration unit, which is used to calculate the full attention score matrix of the offline calibration sequence, the first sparse attention score matrix in the spatial head sparse mode, and the second sparse attention score matrix in the temporal head sparse mode through the first attention head; based on the full attention score matrix and the first sparse attention score matrix of the offline calibration sequence, calculate the first attention recall corresponding to the spatial head sparse mode; and based on the full attention score matrix and the second sparse attention score matrix of the offline calibration sequence, calculate the second attention recall corresponding to the temporal head sparse mode.

[0154] Optionally, the sparse pattern acquisition module 1301 further includes a storage unit for storing the sparse patterns of at least one attention head; correspondingly, the sparse processing module 1302 is specifically used to read the stored sparse patterns of at least one attention head in the t-th time step, and perform attention sparse processing on the first sequence through the first transformation layer based on the read sparse patterns of at least one attention head.

[0155] Optionally, the target sequence includes a first sampling sequence and a second sampling sequence corresponding to the first sequence. The first sampling sequence is obtained by sampling the first sequence at intervals, and the second sampling sequence is a sequence segment in the first sequence. Based on this, the sparse pattern acquisition module 1301 includes a sparse pattern online recognition unit, which is used to calculate the first attention recall rate of the first sampling sequence in the spatial head sparse mode and the second attention recall rate of the second sampling sequence in the temporal head sparse mode at the t-th time step using a first attention head.

[0156] Optionally, the sparse pattern acquisition module 1301 further includes a storage unit for caching the sparse patterns of at least one attention head of the first transformation layer determined in the t-th time step; correspondingly, the sparse processing module 1302 is used to acquire the cached sparse patterns of at least one attention head of the first transformation layer in the t+1-th time step, and perform attention sparse processing on the input sequence through the first transformation layer based on the sparse patterns of at least one attention head.

[0157] Optionally, the sparse pattern acquisition module 1301 is further configured to: if at least one of the first attention recall rate and the second attention recall rate is greater than the recall rate threshold, then the sparse pattern corresponding to the maximum recall rate among the at least one attention recall rate is determined as the sparse pattern of the first attention head.

[0158] Optionally, the sparse pattern acquisition module 1301 is further configured to: if the first attention recall rate and the second attention recall rate are both not greater than the recall rate threshold, then determine that the sparse pattern of the first attention head is the texture head sparse pattern.

[0159] Optionally, the texture head sparse mode includes a first sparse mode and a second sparse mode; the first sparse mode is used to instruct the corresponding attention head to sample the key matrix and value matrix of the input sequence according to preset sampling parameters to obtain a sparse key matrix and a sparse value matrix, and to calculate the attention of the input sequence based on the sparse key matrix and the sparse value matrix; the second sparse mode is used to instruct the corresponding attention head to calculate the corresponding partial attention based on the partial query vectors in the query matrix of the input sequence in the current time step, and to obtain the partial attention corresponding to the partial query vectors in the query matrix of the input sequence calculated in the target time step before the current time step, and to determine the attention of the input sequence in the current time step based on the partial attention calculated by itself and the partial attention obtained in the target time step.

[0160] Optionally, the sparse pattern acquisition module 1301 is further configured to: if the current time step is the i-th time step, determine the sparse pattern of the first attention head as the second sparse pattern, i being greater than the first value and less than the second value, the first value being greater than or equal to 1, and the second value being less than or equal to the total number of time steps required for inference of the target model; if the current time step is the j-th time step, determine the sparse pattern of the first attention head as the first sparse pattern, j being not greater than the first value or not less than the second value.

[0161] Optionally, the difference in sparsity between any two sparsity modes among the multiple sparsity modes is less than the sparsity difference threshold.

[0162] Optionally, the sparse pattern acquisition module 1301 is specifically used to: acquire multiple sparse pattern groups, each sparse pattern group including multiple sparse patterns, wherein the sparsity difference between any two sparse patterns in a sparse pattern group is less than a sparsity difference threshold, and the sparsity difference between any two sparse patterns in different sparse pattern groups is not less than a sparsity difference threshold.

[0163] Optionally, the sparse pattern acquisition module 1301 is further configured to determine the sparse pattern with the largest sparsity among the first attention recall rates corresponding to the multiple spatial head sparse patterns in the multiple sparse pattern groups and the second attention recall rates corresponding to the multiple temporal head sparse patterns as the sparse pattern of the first attention head if at least one attention recall rate is greater than the recall rate threshold.

[0164] Optionally, the first sequence is a sequence corresponding to multiple video frames included in the video, or the first sequence is a sequence corresponding to multiple image regions included in the image.

[0165] In this embodiment, for any attention head in a transform layer of the model, a sparse mode of the attention head is determined, and then attention sparse processing is performed on the first input sequence based on the sparse mode of the attention head. The sparse mode is determined according to the attention head mode of the attention head, which indicates how the attention head focuses on the input sequence; that is, it indicates which aspect of the dependencies in the input sequence the attention head focuses on. Therefore, in this embodiment, for different attention heads in the model, a corresponding sparse mode is used for attention sparse processing based on how the attention head focuses on the input sequence. This can reduce computational overhead and shorten computation time while ensuring computational accuracy as much as possible, thereby better supporting the generation of higher quality and longer duration videos or high-quality images.

[0166] It should be noted that the module division in the data processing apparatus provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0167] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a terminal device, server, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] Furthermore, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0170] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for ease of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0171] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0172] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Obtain the sparse pattern of at least one attention head in the first transformation layer of the target model, wherein the sparse pattern is used to indicate the manner of attention sparse processing through the attention head, and the sparse pattern is determined based on the attention head pattern of the attention head; Based on the sparse pattern of the at least one attention head, the first sequence to be processed is subjected to attention sparse processing through the first transformation layer.

2. The method according to claim 1, characterized in that, The attention head modes include spatial head mode, temporal head mode, and texture head mode. The spatial head mode refers to the mode in which the attention head focuses on the dependencies between multiple subsequences in the input sequence and the dependencies between elements within each subsequence. The temporal head mode refers to the mode in which the attention head focuses on the relationships between sequence segments at the same position in multiple subsequences in the input sequence. The texture head mode refers to modes other than the spatial head mode and the temporal head mode.

3. The method according to claim 2, characterized in that, The step of obtaining the sparse pattern of at least one attention head of the first transformation layer in the target model includes: Multiple sparse modes are obtained, including the spatial header sparse mode corresponding to the spatial header mode, the temporal header sparse mode corresponding to the temporal header mode, and the texture header sparse mode corresponding to the texture header mode. The first attention recall rate of the target sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode are calculated using the first attention head in the at least one attention head. The attention recall rate of the target sequence in any sparse mode is used to indicate the accuracy loss of the target sequence in the corresponding sparse mode of sparse attention relative to full attention. Based on the first attention recall rate and the second attention recall rate, the sparse pattern of the first attention head is determined.

4. The method according to claim 3, characterized in that, The target sequence is an offline calibrated sequence, which is different from the first sequence. The first attentional recall rate of the target sequence in the spatial head sparse mode and the second attentional recall rate in the temporal head sparse mode are calculated using the first attentional head in at least one attentional head, including: The full attention score matrix of the offline calibration sequence, the first sparse attention score matrix in the spatial head sparse mode, and the second sparse attention score matrix in the temporal head sparse mode are calculated using the first attention head. Based on the full attention score matrix of the offline calibration sequence and the first sparse attention score matrix, calculate the first attention recall corresponding to the spatial head sparse pattern; based on the full attention score matrix of the offline calibration sequence and the second sparse attention score matrix, calculate the second attention recall corresponding to the temporal head sparse pattern.

5. The method according to claim 3 or 4, characterized in that, After obtaining the sparse pattern of at least one attention head of the first transformation layer in the target model, the method further includes: Store the sparse pattern of the at least one attention head; The sparse pattern based on the at least one attention head, through the first transform layer, performs attention sparse processing on the first sequence to be processed, including: At the t-th time step, the sparse pattern of the at least one attention head stored is read, and based on the read sparse pattern of the at least one attention head, the first sequence is subjected to attention sparse processing through the first transformation layer.

6. The method according to claim 3, characterized in that, The target sequence includes a first sampling sequence and a second sampling sequence corresponding to the first sequence. The first sampling sequence is obtained by sampling the first sequence at intervals, and the second sampling sequence is a sequence segment of the first sequence. The step of calculating the first attention recall rate of the target sequence in the spatial head sparse mode and the second attention recall rate in the temporal head sparse mode using the first attention head in at least one attention head includes: At time step t, the first attention recall rate of the first sampled sequence in the spatial head sparse mode is calculated using the first attention head, and the second attention recall rate of the second sampled sequence in the temporal head sparse mode is calculated.

7. The method according to claim 6, characterized in that, The method further includes: Cache the sparse pattern of at least one attention head of the first transform layer determined at time step t; At time step t+1, the sparse pattern of at least one attention head of the first transform layer in the cache is obtained, and based on the sparse pattern of the at least one attention head, the input sequence is subjected to attention sparse processing through the first transform layer.

8. The method according to any one of claims 3 to 7, characterized in that, Determining the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate includes: If at least one of the first attention recall rate and the second attention recall rate is greater than the recall rate threshold, then the sparse pattern corresponding to the maximum recall rate among the at least one attention recall rate is determined as the sparse pattern of the first attention head.

9. The method according to any one of claims 3 to 8, characterized in that, The method further includes: If both the first attention recall rate and the second attention recall rate are not greater than the recall rate threshold, then the sparse mode of the first attention head is determined to be the sparse mode of the texture head.

10. The method according to claim 9, characterized in that, The texture head sparse mode includes a first sparse mode and a second sparse mode; The first sparse mode is used to instruct the corresponding attention head to sample the key matrix and value matrix of the input sequence according to preset sampling parameters to obtain a sparse key matrix and a sparse value matrix, and to calculate the attention of the input sequence based on the sparse key matrix and the sparse value matrix; The second sparse mode is used to instruct the corresponding attention head to calculate the corresponding partial attention based on the partial query vector in the query matrix of the input sequence in the current time step, and to obtain the partial attention corresponding to the partial query vector in the query matrix of the input sequence calculated in the target time step before the current time step, and to determine the attention of the input sequence in the current time step based on the partial attention calculated by itself and the partial attention obtained in the target time step.

11. The method according to claim 10, characterized in that, Determining the sparse mode of the first attention head as the sparse mode of the texture head includes: If the current time step is the i-th time step, then the sparse mode of the first attention head is determined to be the second sparse mode, where i is greater than a first value and less than a second value, the first value is greater than or equal to 1, and the second value is less than or equal to the total number of time steps required for the inference of the target model. If the current time step is the j-th time step, then the sparse mode of the first attention head is determined to be the first sparse mode, where j is not greater than the first value or not less than the second value.

12. The method according to any one of claims 3 to 11, characterized in that, The difference in sparsity between any two sparsity modes in the plurality of sparsity modes is less than the sparsity difference threshold.

13. The method according to any one of claims 3 to 7, characterized in that, The acquisition of multiple sparse patterns includes: Multiple sparse pattern groups are obtained, each sparse pattern group includes the multiple sparse patterns, wherein the sparsity difference between any two sparse patterns in a sparse pattern group is less than the sparsity difference threshold, and the sparsity difference between any two sparse patterns in different sparse pattern groups is not less than the sparsity difference threshold.

14. The method according to claim 13, characterized in that, Determining the sparse pattern of the first attention head based on the first attention recall rate and the second attention recall rate includes: If at least one of the first attention recall rates corresponding to the multiple spatial head sparse patterns in the multiple sparse pattern groups and the second attention recall rates corresponding to the multiple temporal head sparse patterns is greater than the recall rate threshold, then the sparse pattern with the largest sparsity among the at least one sparse patterns corresponding to the at least one attention recall rate is determined as the sparse pattern of the first attention head.

15. The method according to any one of claims 1 to 14, characterized in that, The first sequence is a sequence corresponding to multiple video frames included in a video, or the first sequence is a sequence corresponding to multiple image regions included in an image.

16. A data processing apparatus, characterized in that, The data processing device includes: A sparse pattern acquisition module is used to acquire the sparse pattern of at least one attention head of the first transformation layer in the target model. The sparse pattern is used to indicate the way attention sparse processing is performed through the attention head, and the sparse pattern is determined based on the attention head pattern of the attention head. A sparse processing module is used to perform attention sparse processing on the first sequence to be processed through the first transformation layer based on the sparse pattern of the at least one attention head.

17. A computing device, characterized in that, The computing device includes a processor for executing at least one program instruction or code stored in a memory to implement the data processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computing device, cause the computing device to perform the data processing method according to any one of claims 1 to 15.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the data processing method as described in any one of claims 1 to 15.

Images