A data processing method, device, system and computer readable storage medium

Abstract

The application discloses a data processing method, device, system and computer readable storage medium, relates to the technical field of data processing, and self-attention calculation and cross-attention calculation are synchronously performed, and the cross-attention calculation does not depend on the result of the self-attention calculation. Through parallel processing and independent calculation, the data processing path of the diffusion model is shortened, the input of the next data block is generated faster, the inference process of the whole diffusion model is accelerated, the technical problem that the inference delay of the diffusion model is high in the related art and it is difficult to meet the real-time requirement is solved, and the beneficial effects of reducing the model inference delay and improving the real-time performance are achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, system and computer-readable storage medium. Background Art

[0002] In a multimodal generation scenario using a diffusion model, in order to generate visual content that is aligned with the semantic text input by the user, during the inference process of the diffusion model, due to constraints such as data dependency, computational logic differences, and model structure, the image self-attention mechanism and text cross-attention mechanism in the diffusion model need to be processed sequentially, resulting in a long data processing link, which in turn lengthens the inference latency and cannot meet real-time requirements.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the Invention

[0004] The present application provides a data processing method, device, system and computer-readable storage medium to at least solve the problem in the related art that the diffusion model inference delay is high and it is difficult to meet the real-time requirements.

[0005] The present application provides a data processing method, which is applied to a computing device deployed with a diffusion model, wherein the diffusion model includes multiple data blocks. The data processing method includes: obtaining an input visual vector, text data, and time vector corresponding to a current data block; using the input visual vector and the time vector to perform self-attention calculation to obtain a first vector; using the input visual vector, the time vector, and the text data to perform cross-attention calculation to obtain a second vector; and calculating the input visual vector of the next data block based on the first vector and the second vector.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data processing methods when executing the computer program.

[0007] The present application also provides a data processing system, including a memory expansion device and at least one electronic device as described above.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0009] Through this application, self-attention calculation and cross-attention calculation are carried out simultaneously, and the cross-attention calculation does not depend on the result of self-attention calculation. Through parallel processing and independent calculation, the data processing path of the diffusion model is shortened, and the input of a data block is generated more quickly, thereby accelerating the reasoning process of the entire diffusion model. It solves the technical problems in related technologies that the diffusion model reasoning delay is high and it is difficult to meet real-time requirements, and achieves the beneficial effect of reducing model reasoning delay and improving real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application.

[0012] Figure 2 A flowchart of the steps of a data processing method provided in an embodiment of the present application.

[0013] Figure 3 A schematic diagram of the basic model architecture and training process of a diffusion model provided in an embodiment of the present application.

[0014] Figure 4 A schematic diagram of the structure of a data block provided in an embodiment of the present application.

[0015] Figure 5 This is a structural diagram of a data block in related technology.

[0016] Figure 6 A schematic diagram of a diffusion model training provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 The data processing system includes a memory expansion device and at least one computing device. The memory expansion device and the computing device can communicate via the CXL (Compute ExpressLink) protocol. The computing device includes but is not limited to a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), etc. The memory expansion device includes but is not limited to an accelerator card based on the CXL protocol.

[0021] The embodiment of the present application provides a data processing method, combined with Figure 2 The execution flow of the data processing method shown in FIG. 1 is described in detail. The method in this embodiment can be implemented by the above-mentioned computing device, which is deployed with a diffusion model. Figure 3 , explains the basic model architecture and training process of the diffusion model, original image Added server generated random noise , the image after noise addition The image is obtained through the diffusion process It can be understood that the diffusion process is a process of gradually adding noise to generate noise data for training. During the training process, the model estimates the noise at that moment through the deep learning model based on the information of the time vector t. The output of the computational model and The loss between the two is calculated, and the gradient of the model parameters is updated according to the loss, thereby optimizing the model's ability to estimate noise. Among them, the noise estimation model uses the Transformer (a deep learning model architecture based on the self-attention mechanism) architecture to form a DiT (Diffusion Transformer) model. The model consists of multiple data blocks (N blocks). The output of each data block is used as the input of the next data block, that is, the output of the i-th data block is used as the i-th data block. The input of the data block ( ).

[0022] S101: Obtain the input visual vector, text data and time vector corresponding to the current data block.

[0023] Assuming that the current data block is the first data block, the input visual vector corresponding to the current data block is the feature vector corresponding to the original image data provided by the user. Assuming that the current data block is not the first data block, the input visual vector corresponding to the current data block is the output of the previous data block. The text data is the semantic guidance input by the user, which is used to guide the model on how to process the original image data. The text data provides semantic information for the diffusion model so that the diffusion model understands the user's needs and generates image data that matches the text description. For example, if the user wants to perform style conversion on the original image data, the input text can be to convert the image to black and white style. If the user wants to edit the content of the image, the input text can be to add a cat to the image. The time vector is a parameter used to control the time step of the diffusion process in the diffusion model. The time vector facilitates the diffusion model to understand the current processing stage and ensures that the diffusion model can correctly estimate the noise at different time steps and gradually generate the target image.

[0024] After obtaining the input visual vector and time vector, the data visual data corresponding to the current data block is layer-normalized to obtain the first input vector. Then, the rotation position encoding is added to the linear calculation result of the first input vector and the time vector to obtain the second input vector. Self-attention calculation and cross-attention calculation are performed on the second input vector in parallel.

[0025] Layer normalization is a normalization technique that stabilizes the training process and improves model performance. By normalizing each feature dimension of the input visual vector to ensure a mean of 0 and a standard deviation of 1, the first input vector is obtained after layer normalization. The time vector and the first input vector are added together, processed through a linear transformation layer, and Rotary Position Embedding (RoPE) is added to obtain the second input vector. Rotary Position Embedding (RoPE) is an improved positional encoding method that better captures positional information in sequence data while avoiding the performance degradation of traditional positional encoding in long sequences.

[0026] S102: Use the input visual vector and time vector to perform self-attention calculation to obtain a first vector.

[0027] In this embodiment, the second input vector is used as the input of the self-attention mechanism, and self-attention calculation is performed on the second input vector. The self-attention mechanism allows the diffusion model to dynamically focus on different parts of the input data when processing the input data. By calculating the feature correlation within the input data, the diffusion model can better capture the global information and local features in the data. The result of the self-attention calculation is the first vector.

[0028] Specifically, such as Figure 4 , the second input vector is subjected to three different linear transformations to generate a query matrix Q, a key matrix K and a value matrix V, and then the first vector is calculated based on the query matrix Q, the key matrix K and the value matrix V. It can be understood that the first vector contains the global and local features of the input visual vector.

[0029] In this embodiment, layer normalization and linear transformation enable more stable processing of input data while better capturing temporal information. The self-attention mechanism can more comprehensively represent the global information and local features of the input visual vector, improving image comprehension. By combining global and local information and introducing temporal information, higher-quality images can be generated.

[0030] S103: Using the input visual vector, time vector, and text data, a cross-attention calculation is performed to obtain a second vector.

[0031] In this embodiment, the second input vector is used as the input of the cross attention mechanism, and the text vector corresponding to the text data is superimposed on the query matrix Q in the cross attention mechanism, that is, , where Q is the query matrix extracted from the second input vector, TV is the text vector, The query matrix after superimposing the text vector can be realized by addition operation, so that the query matrix contains both visual features and text semantic information. The second vector is obtained by performing a cross-attention calculation with the key matrix K and the value matrix V, which are extracted from the second input vector. It can be understood that the second vector combines the features of the input visual vector and the semantic information of the text data, enabling the model to perform targeted processing on the image based on the text semantics.

[0032] S104: Calculate the input visual vector of the next data block based on the first vector and the second vector.

[0033] In this embodiment, after the two attention calculations are completed, a vector concatenation operation is performed on the first and second vectors based on an MLP (Multi-Layer Perceptron). This is then residually connected with the input visual vector, and finally enters the FFN (Feed-Forward Network) to generate the input visual vector for the next data block. By concatenating the first and second vectors through the MLP, the model can more comprehensively integrate the results of self-attention and cross-attention, improving feature representation capabilities. Further processing by the FFN can better transform and optimize features, generating higher-quality input visual vectors.

[0034] Through this application, self-attention calculation and cross-attention calculation are carried out simultaneously, and the cross-attention calculation does not depend on the result of self-attention calculation. Through parallel processing and independent calculation, the data processing path of the diffusion model is shortened, and the input of a data block is generated more quickly, thereby accelerating the reasoning process of the entire diffusion model. It solves the technical problems in related technologies that the diffusion model reasoning delay is high and it is difficult to meet real-time requirements, and achieves the beneficial effect of reducing model reasoning delay and improving real-time performance.

[0035] In an exemplary embodiment, the input visual vector of the next data block is calculated based on the first vector and the second vector, including: performing a nonlinear transformation on the first vector to obtain a third vector; performing a nonlinear transformation on the second vector to obtain a fourth vector; and calculating the input visual vector of the next data block based on a fifth vector obtained by concatenating the third vector and the fourth vector.

[0036] In this embodiment, the first vector is input into the MLP for nonlinear transformation to obtain a third vector, the second vector is input into the MLP for nonlinear transformation to obtain a fourth vector, and then the third vector and the fourth vector are spliced ​​to obtain a fifth vector, and the fifth vector is used to calculate the input visual vector of the next data block, where the splicing operation can be a simple vector splicing or a more complex feature fusion operation.

[0037] The first and second vectors are first fed into the MLP for nonlinear transformation, independently extracting the complex features of each vector. The MLP outputs are then concatenated. Because the features of each vector are processed independently before concatenation, any confusion of feature information that may occur during the concatenation process is avoided. This allows the features of each vector to maintain their uniqueness, and the concatenated feature vector can better fuse the two types of information. The MLP's nonlinear transformation provides an independent gradient path for each vector, avoiding the vanishing or exploding gradients that may occur during the concatenation process. Furthermore, the MLP's nonlinear transformation reduces the dimensionality of each vector, making the concatenated feature vector more compact and reducing the computational resources consumed by the computing device.

[0038] Of course, the first vector and the second vector may be concatenated first, and the concatenated vector may be input into the MLP for nonlinear calculation to obtain a fifth vector, so as to calculate the input visual vector of the next data block based on the fifth vector.

[0039] In an exemplary embodiment, the input visual vector of the next data block is calculated based on the fifth vector obtained by concatenating the third vector and the fourth vector, including: obtaining a residual feature vector based on the input visual vector corresponding to the current data block and the fifth vector obtained by concatenating the third vector and the fourth vector; and performing a linear transformation on the residual feature vector to obtain the input visual vector of the next data block.

[0040] In this embodiment, refer to Figure 4 As shown in the figure, after obtaining the fifth vector, the fifth vector is residually connected to the input visual vector to obtain a residual feature vector. A forward linear calculation is performed on the residual feature vector to generate the input visual vector of the next data block. Specifically, the residual feature vector is input to the normalization layer for processing and then output through the FFN. The vector output by the FFN is residually connected with the residual feature vector to obtain the input visual vector of the next data block.

[0041] Compared with the solution of this embodiment Figure 5 The solutions in the related technologies shown significantly shorten the computational processing flow, reduce calculations such as residual connections, and can reduce the model inference latency by one third. Figure 4 and Figure 5 The feedforward networks in can all be FNNs.

[0042] In an exemplary embodiment, the data processing method also includes: writing the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block into a memory expansion device connected to the computing device; before using the input visual vector and the time vector to perform self-attention calculation to obtain the first vector, the data processing method also includes: writing all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device; using the input visual vector and the time vector to perform self-attention calculation to obtain the first vector, including: using the input visual vector, the time vector, all the first intermediate parameters stored in the storage medium, the first intermediate parameters in the self-attention calculation process of the current data block, and the self-attention matrix to calculate the first vector; using the input visual vector, the time vector, and the text data to perform cross-attention calculation to obtain the second vector, including: using the input visual vector, the time vector, the text data, all the second intermediate parameters stored in the storage medium, the second intermediate parameters in the cross-attention calculation process of the current data block, and the cross-attention matrix to calculate the second vector.

[0043] In this embodiment, combined with Figure 1 , the calculation process of one Block is explained, and the same applies to other Blocks. The computing device reads the input visual vector and text data of the first Block from its own storage medium (such as video memory) to perform self-attention and cross-attention calculations. When the two attention calculations are completed, the computing device sends CXL Flit (ComputeExpress Link Flow Control Unit Information) information to the CXL-based KVcache manager in the memory expansion device, and the memory expansion device initiates instructions to start reading and saving self-attention (That is, the first intermediate parameter in this embodiment, including the key matrix Sum Matrix ) and cross-attention (That is, the second intermediate parameter in this embodiment, including the key matrix Sum Matrix ). At this time, the computing device is performing subsequent feature concatenation, FFN, and other calculations. It should be noted that the two attention calculations are synchronized, while the corresponding KVcache storage is not synchronized. This is because the reading bandwidth of large amounts of data is limited, especially during long sequence inference. While the KVcache storage is in progress, the computing device is normally performing subsequent calculations, thus achieving mutual masking of computation and communication.

[0044] The computing device completes the calculation of the Block, passes it to the loop of the second Block, and sends CXL Flit information to notify the memory expansion device that it can start the write operation. At this time, the computing device is injecting the time vector and RoPE. Due to the use of KVcacahe acceleration technology, the memory expansion device will and Write to the storage medium of the computing device so that the computing device can combine the first Block’s cross attention and self attention calculation process corresponding to the second Block and To calculate the first vector and second vector corresponding to the second Block respectively.

[0045] As an optional embodiment, for each data block, the corresponding vector of the data block is extracted from the second input vector of the data block. and , then when calculating the first vector and the second vector, based on the corresponding and , and all data blocks before this data block obtained from the storage medium and Generate the first vector and the second vector. Reuse the previous and The amount of calculation can be further reduced.

[0046] In this embodiment, during long-sequence reasoning, self-attention and cross-attention calculations generate a large number of intermediate parameters (such as the key matrix K and the value matrix V). If all these parameters were stored in the computing device's internal storage medium (such as video memory), this could lead to insufficient video memory and limit the model's processing power. By storing these intermediate parameters in a memory expansion device, the computing device's memory pressure can be effectively reduced, enabling it to process longer sequences or larger models. While the computing device is performing self-attention and cross-attention calculations, the memory expansion device can read and save the intermediate parameters (KV cache) in parallel. This parallel operation allows computation and communication to overlap, reducing latency and improving overall computational efficiency. For example, while the computing device is performing feature concatenation and FFN calculations, the memory expansion device can simultaneously save the intermediate parameters of the current block without affecting the normal operation of the computing device. By storing these intermediate parameters (KV cache), these parameters can be directly reused in subsequent block calculations without recalculation, reducing the overhead of repeated calculations. This significantly improves computational efficiency, especially in long-sequence reasoning. In addition, while the memory expansion device is reading and saving data, the computing device can continue to perform subsequent computing tasks, such as feature splicing and FFN. This parallel operation makes full use of the computing device's resources and improves overall computing efficiency.

[0047] In an exemplary embodiment, the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block are written into a memory expansion device connected to the computing device, including: writing the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block into the memory expansion device connected to the computing device in a first writing order; writing all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device, including: writing all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device in a second writing order; the first writing order and the second writing order are opposite.

[0048] In this embodiment, the first write data is a specific order when writing the intermediate parameters (KVcache) of the current data block to the memory expansion device, and the second write data is a specific order when writing the intermediate parameters stored in the memory expansion device to the internal storage medium of the computing device.

[0049] In the and Write to the memory expansion device, if the first write order is to write Rewrite , then the second write order is to write first Rewrite , then in and When writing to the storage medium of a computing device, first write Rewrite By optimizing the writing and reading order of KVcache, data in the cache line is read and written first, which reduces the latency of video memory access and improves the inference speed of the model.

[0050] In an exemplary embodiment, the diffusion model includes a self-attention model and a cross-attention model; the data processing method also includes: iteratively training the self-attention model; when the number of iterations reaches a preset number, training the self-attention model and the cross-attention model according to a first set of learning rate ratios; the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the first set of learning rate ratios is greater than a preset value; training the self-attention model and the cross-attention model according to a second set of learning rate ratios, and the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the second set of learning rate ratios is less than a preset value, and stopping training until the diffusion model meets the training end conditions.

[0051] In this embodiment, combined with Figure 6 The training of the diffusion model consists of three stages. The first stage trains the self-attention model without using text injection. It mainly trains the image generation quality and trains for about 100 rounds (depending on the size of the training data, such as 100 rounds for every 100,000 data). By training the self-attention model alone, the model can better capture the global information and local features of the image, thereby generating higher quality images. In the second stage, the learning rate ratio of the self-attention module parameters is adjusted to , the cross attention learning rate ratio is In this stage, the main training is the text information guidance brought by cross attention. The learning rate ratio here can be adjusted, but the learning rate ratio of the self-attention branch cannot be set to 0 to avoid divergence of the loss value. By adjusting the learning rate ratio, the model can more effectively learn the guiding role of text information in image generation, making the generated image more consistent with the text description. The third stage is collaborative training, and the learning rate ratio is set to and (or two similar values). The learning rate is adjusted here to accelerate the rapid convergence of the calculation process subsequent to the self-attention module. Through collaborative training, the model can better integrate the results of self-attention and cross-attention, improving the overall generation quality. This staged training method can significantly improve training efficiency and avoid processing complex tasks simultaneously in the early training stages, thereby reducing training time and computing resource consumption. By properly adjusting the learning rate, the model can converge faster and improve training efficiency. This is because the two attention branches here are independently calculated and have no intersection. In the third stage, the model is trained collaboratively, and the learning rate ratio is set to a similar value. The main training parameters in this stage are concentrated in the MLP layer, because the MLP layer is responsible for fusing the results of the two attention modules. By optimizing the parameters of the MLP layer, the model can better integrate the results of self-attention and cross-attention, improving the overall generation quality.

[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0053] An embodiment of the present application also provides a data processing device, which is applied to a computing device deployed with a diffusion model, the diffusion model includes multiple data blocks, and the data processing device includes: a first acquisition module, used to obtain the input visual vector, text data and time vector corresponding to the current data block; a first calculation module, used to use the input visual vector and time vector to perform self-attention calculation to obtain a first vector; a second calculation module, used to use the input visual vector, time vector, and text data to perform cross-attention calculation to obtain a second vector; and a third calculation module, used to calculate the input visual vector of the next data block based on the first vector and the second vector.

[0054] In an exemplary embodiment, the input visual vector of the next data block is calculated based on the first vector and the second vector, including: performing a nonlinear transformation on the first vector to obtain a third vector; performing a nonlinear transformation on the second vector to obtain a fourth vector; and calculating the input visual vector of the next data block based on a fifth vector obtained by concatenating the third vector and the fourth vector.

[0055] In an exemplary embodiment, the input visual vector of the next data block is calculated based on the fifth vector obtained by concatenating the third vector and the fourth vector, including: obtaining a residual feature vector based on the input visual vector corresponding to the current data block and the fifth vector obtained by concatenating the third vector and the fourth vector; and performing a linear transformation on the residual feature vector to obtain the input visual vector of the next data block.

[0056] In an exemplary embodiment, after obtaining the input visual vector, text data and time vector corresponding to the current data block, the data processing device is also used to: perform layer normalization on the data visual data corresponding to the current data block to obtain a first input vector; add rotation position encoding to the linear calculation result of the first input vector and the time vector to obtain a second input vector; use the input visual vector and the time vector to perform self-attention calculation to obtain the first vector, including: perform self-attention calculation on the second input vector to obtain the first vector; use the input visual vector, the time vector, and the text data to perform cross-attention calculation to obtain the second vector, including: use the second input vector and the text data to perform cross-attention calculation to obtain the second vector.

[0057] In an exemplary embodiment, the data processing device is also used to: write the first intermediate parameter in the self-attention calculation process of the current data block, and the second intermediate parameter in the cross-attention calculation process of the current data block, into a memory expansion device connected to the computing device; before using the input visual vector and the time vector to perform self-attention calculation to obtain the first vector, the data processing device is also used to: write all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device; use the input visual vector and the time vector to perform self-attention calculation to obtain the first vector, including: using the input visual vector, the time vector, all the first intermediate parameters stored in the storage medium, the first intermediate parameters in the self-attention calculation process of the current data block, and the self-attention matrix to calculate the first vector; use the input visual vector, the time vector, and the text data to perform cross-attention calculation to obtain the second vector, including: using the input visual vector, the time vector, the text data, all the second intermediate parameters stored in the storage medium, the second intermediate parameters in the cross-attention calculation process of the current data block, and the cross-attention matrix to calculate the second vector.

[0058] In an exemplary embodiment, the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block are written into a memory expansion device connected to the computing device, including: writing the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block into the memory expansion device connected to the computing device in a first writing order; writing all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device, including: writing all the first intermediate parameters and all the second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device in a second writing order; the first writing order and the second writing order are opposite.

[0059] In an exemplary embodiment, the diffusion model includes a self-attention model and a cross-attention model; the data processing method also includes a training module configured to iteratively train the self-attention model; when the number of iterations reaches a preset number, the self-attention model and the cross-attention model are trained according to a first set of learning rate ratios; the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the first set of learning rate ratios is greater than a preset value; the self-attention model and the cross-attention model are trained according to a second set of learning rate ratios, and the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the second set of learning rate ratios is less than a preset value, and training is stopped until the diffusion model meets the training end condition.

[0060] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, and no further details will be given here.

[0061] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data processing method embodiments.

[0062] An embodiment of the present application further provides a data processing system, comprising a memory expansion device and at least one electronic device as described above.

[0063] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when run.

[0064] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0065] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above data processing method embodiments are implemented.

[0066] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data processing method embodiments are implemented.

[0067] For an introduction to an electronic device, a data processing system, a computer-readable storage medium, and a computer program product provided in this application, please refer to the above embodiments, and this application will not go into details here.

[0068] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] The above is a detailed introduction to a data processing method, device, system and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: Applied to a computing device deployed with a diffusion model, the diffusion model includes multiple data blocks, and the data processing method includes: Get the input visual vector, text data and time vector corresponding to the current data block; Performing self-attention calculation using the input visual vector and the time vector to obtain a first vector; Performing a cross-attention calculation using the input visual vector, the time vector, and the text data to obtain a second vector; Calculate an input visual vector of a next data block based on the first vector and the second vector; The data processing method further includes: Writing the first intermediate parameter in the self-attention calculation process of the current data block and the second intermediate parameter in the cross-attention calculation process of the current data block into a memory expansion device connected to the computing device; Before performing self-attention calculation using the input visual vector and the time vector to obtain the first vector, the data processing method further includes: Writing all first intermediate parameters and all second intermediate parameters stored in the memory expansion device into a storage medium inside the computing device; Performing self-attention calculation using the input visual vector and the time vector to obtain a first vector includes: Calculating a first vector using the input visual vector, the time vector, all first intermediate parameters stored in the storage medium, the first intermediate parameter in the self-attention calculation process of the current data block, and a self-attention matrix; Performing a cross-attention calculation using the input visual vector, the time vector, and the text data to obtain a second vector includes: The second vector is calculated using the input visual vector, the time vector, the text data, all second intermediate parameters stored in the storage medium, the second intermediate parameters in the cross-attention calculation process of the current data block, and the cross-attention matrix.

2. The data processing method according to claim 1, wherein: Calculating an input visual vector of a next data block based on the first vector and the second vector includes: Performing a nonlinear transformation on the first vector to obtain a third vector; Performing a nonlinear transformation on the second vector to obtain a fourth vector; An input visual vector of a next data block is calculated based on a fifth vector obtained by concatenating the third vector and the fourth vector.

3. The data processing method according to claim 2, characterized in that: Calculating an input visual vector of a next data block based on a fifth vector obtained by concatenating the third vector and the fourth vector includes: Obtaining a residual feature vector based on an input visual vector corresponding to the current data block and a fifth vector obtained by concatenating the third vector and the fourth vector; Performing a linear transformation on the residual feature vector to obtain an input visual vector of the next data block.

4. The data processing method according to claim 1, wherein: After obtaining the input visual vector, text data, and time vector corresponding to the current data block, the data processing method further includes: Perform layer normalization on the data visual data corresponding to the current data block to obtain a first input vector; Adding a rotational position code to a linear calculation result of the first input vector and the time vector to obtain a second input vector; Performing self-attention calculation using the input visual vector and the time vector to obtain a first vector includes: Performing self-attention calculation on the second input vector to obtain a first vector; Performing a cross-attention calculation using the input visual vector, the time vector, and the text data to obtain a second vector includes: A second vector is obtained by performing a cross-attention calculation using the second input vector and the text data.

5. The data processing method according to claim 1, wherein: Writing a first intermediate parameter in a self-attention calculation process of the current data block and a second intermediate parameter in a cross-attention calculation process of the current data block into a memory expansion device connected to the computing device, comprising: Writing, in a first writing order, a first intermediate parameter in a self-attention calculation process of the current data block and a second intermediate parameter in a cross-attention calculation process of the current data block into a memory expansion device connected to the computing device; Writing all first intermediate parameters and all second intermediate parameters stored in the memory expansion device into a storage medium inside the computing device includes: Writing all first intermediate parameters and all second intermediate parameters stored in the memory expansion device into the storage medium inside the computing device in sequence according to a second writing order; The first writing order is opposite to the second writing order.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The diffusion model includes a self-attention model and a cross-attention model; The data processing method further includes: Iteratively training the self-attention model; When the number of iterations reaches a preset number, the self-attention model and the cross-attention model are trained according to a first set of learning rate ratios; the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the first set of learning rate ratios is greater than a preset value; The self-attention model and the cross-attention model are trained according to a second set of learning rate ratios, wherein the difference between the learning rate ratio of the self-attention model and the learning rate ratio of the cross-attention model in the second set of learning rate ratios is less than the preset value, and the training is stopped until the diffusion model meets the training end condition.

7. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 6 when executing the computer program.

8. A data processing system, characterized in that: The electronic device comprises a memory expansion device and at least one electronic device as claimed in claim 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images