A method, system, device and storage medium for constructing a diffusion transformer model

Abstract

The present invention provides a method, system, device and storage medium for constructing a diffusion transformer model, comprising the following steps: generating corresponding keys and values ​​through a self-attention layer in a diffusion transformer model to be generated; reordering the keys and values ​​into dimensions of image and video features to obtain a key-value pair feature map; compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map; training the diffusion transformer model to be generated using the compressed key-value pair feature map to obtain the diffusion transformer model; the application optimizes the self-attention mechanism of the diffusion transformer model, effectively reducing the computational overhead of calculating the self-attention, and can perform adaptive compression according to the relationship between different spatial positions, retain fine spatial detail information, and improve the generation performance in large-resolution image and video generation tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method, system, device and storage medium for constructing a diffusion transformer model. Background Art

[0002] Inspired by the great success of the Transformer model in natural language processing tasks, the Diffusion Model has received extensive attention and application in the field of computer vision. Among them, the Diffusion Transformer model DiT (Diffusion Transformer) introduces the self-attention mechanism into the diffusion process, showing excellent large-resolution image generation and video generation capabilities; however, the DiT model generates a large number of key-value pairs when processing high-resolution images and videos, resulting in the need to consume huge computing resources to calculate attention, which not only increases the computational complexity of the model, but also greatly limits its practical application on terminal devices. Therefore, effectively compressing key-value pairs and reducing the computational overhead of calculating attention have become key challenges to improve the efficiency of the DiT model.

[0003] In the existing related technologies, deformable convolution is an adaptive convolutional neural network that can enhance the spatial transformation modeling ability of the model by learning the displacement of the convolution kernel. In the traditional convolution model, deformable convolution can be applied to extract image feature maps. However, this method does not take into account the Transformer model, and it does not optimize the self-attention mechanism of the Transformer model. At the same time, the traditional deformable convolution can only process two-dimensional image features and cannot process video generation tasks with a time dimension. Therefore, it cannot effectively reduce the computational overhead of calculating self-attention in the DiT model; PixArt-Sigma's convolutional key value compression adopts an efficient key value compression scheme based on convolution, which compresses key values ​​and greatly reduces storage and computational overhead. However, this method uses the same processing for key values ​​at different positions, without considering the different relationships in spatial positions, resulting in the inability to retain fine spatial detail information, which will cause a decrease in network accuracy; therefore, the present application anticipates a method for constructing a diffusion transformer model. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a method, system, device and storage medium for constructing a diffusion transformer model, so as to solve the technical problem that the network accuracy decreases due to the inability to retain fine spatial detail information in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a diffusion transformer model, comprising the following steps:

[0006] Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value;

[0007] Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0008] Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0009] The method of compressing the key-value pair feature map in each self-attention layer by using a deformable convolution kernel to obtain a compressed key-value pair feature map includes:

[0010] At each position, an additional convolution branch is used to predict the offset in the area around the corresponding position;

[0011] The features at adjacent positions are weighted sampled according to the offset to obtain a compressed key-value pair feature map.

[0012] The diffusion transformer model to be generated is trained using the compressed key-value pair feature map to obtain the diffusion transformer model.

[0013] Further, if the diffusion transformer model is used to generate an image, the offset includes an x ​​offset and a y offset, and the image video feature includes an image two-dimensional feature.

[0014] Further, if the diffusion transformer model is used to generate a video, the offset includes an x ​​offset, a y offset, and a t offset, and the image video feature includes a video three-dimensional feature.

[0015] Furthermore, weighted sampling is performed on features at adjacent positions according to the offset, including:

[0016] Adjust the sampling position of the convolution kernel according to the predicted offset;

[0017] The convolution kernel after adjusting the sampling position is weighted summed with the elements at the corresponding position of the input feature map.

[0018] Furthermore, the weighted sampling process adopts a weighted method of deformable convolution, and the weighted method of deformable convolution includes offset learning, sampling position adjustment, weight allocation and weighted summation in the weighted process.

[0019] In a second aspect, an embodiment of the present invention provides an inference method of a diffusion transformer model constructed by using the method for constructing a diffusion transformer model, comprising:

[0020] Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value;

[0021] Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0022] Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0023] The compressed key-value pair feature map is input into the subsequent module of the diffusion transformer model to obtain an image or video corresponding to the word unit.

[0024] In an embodiment of the present invention, in a third aspect, an embodiment of the present invention provides a system for constructing a diffusion transformer model, including:

[0025] Initialization unit, configured as:

[0026] Used to generate corresponding keys and values ​​by passing the word unit through the self-attention layer in the diffusion transformer model to be generated;

[0027] The processing unit is configured as follows:

[0028] Used to reorder the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0029] Compression units are configured as follows:

[0030] Used to compress the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0031] The building blocks are configured as follows:

[0032] Used to use the compressed key-value pair feature graph to infer the diffusion transformer model to be generated, so as to obtain the diffusion transformer model.

[0033] In an embodiment of the present invention, in a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method or reasoning method for constructing a diffusion transformer model.

[0034] In an embodiment of the present invention, in a fifth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the method or reasoning method for constructing a diffusion transformer model.

[0035] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document.

[0036] Compared with the prior art, the present invention has the following beneficial technical effects:

[0037] The present invention provides a method, system, device and storage medium for constructing a diffusion transformer model, comprising the following steps: generating corresponding keys and values ​​through a self-attention layer in a diffusion transformer model to be generated; reordering the keys and values ​​into dimensions of image and video features to obtain a key-value pair feature map; compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map; training the diffusion transformer model to be generated using the compressed key-value pair feature map to obtain the diffusion transformer model; the present application optimizes the self-attention mechanism of the diffusion transformer model, effectively reducing the computational overhead of calculating the self-attention, and at the same time, the present application can perform adaptive compression according to the relationship between different spatial positions, retain fine spatial detail information, avoid information loss, and improve the generation performance of the diffusion transformer model in large-resolution image and video generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0039] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0040] Figure 1 A flow chart of a method for constructing a diffusion transformer model according to an embodiment of the present disclosure is shown;

[0041] Figure 2A system structure diagram for constructing a diffusion transformer model according to an embodiment of the present disclosure is shown;

[0042] Figure 3 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0044] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0047] Figure 1 A method 100 for constructing a diffusion transformer model according to an embodiment of the present disclosure is shown. The embodiment of the present disclosure includes the following steps:

[0048] At step S101, the word element is passed through the self-attention layer in the diffusion transformer model to be generated to generate corresponding keys and values;

[0049] Specifically, the process is:

[0050] Load the token into the diffusion transformer model for preprocessing, convert the token format based on the requirements of the diffusion transformer model and the self-attention layer, and perform normalization or standardization.

[0051] The preprocessed tokens are forward propagated in the diffusion transformer model and diffused, which is a process to introduce noise and gradually blur the input data;

[0052] The input word usually passes through the embedding layer to convert it into a higher-dimensional vector representation. In the self-attention layer, the model calculates the correlation between different positions in the input data. For each position in the input data, the model generates a corresponding key and value vector. These key and value vectors are used to measure the correlation between different positions in the input and are used for subsequent attention weight calculations.

[0053] Next, go to step S102;

[0054] At step S102,

[0055] Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0056] Further, in an embodiment of the present disclosure, the image and video features (or may be generated object features) include two-dimensional image features and three-dimensional video features, that is, images or videos are generated based on words. Specifically, if the diffusion transformer model is used to generate a target image, the offset includes an x ​​offset and a y offset, and the image and video features include two-dimensional image features; if the diffusion transformer model is used to generate a target video, the offset includes an x ​​offset, a y offset, and a t offset, and the image and video features include three-dimensional video features; specifically, the t offset is a time offset, and by utilizing a time offset model, the timing information in the data can be more effectively captured and utilized.

[0057] In the disclosed embodiment, the key-value pair feature map is used to be input into other subsequent modules (e.g., a self-attention calculation module) in the diffusion transformer model; the key-value pair feature map is input into the subsequent self-attention calculation module to calculate the attention weight and generate the output. Specifically, the key (Key) is used to measure the correlation between different positions, and the value (Value) contains the feature information related to each position. The self-attention calculation module calculates the attention score based on the key-value pair feature map, and performs weighted summation of the values ​​based on these scores to generate the output representation of each position; at the same time, by introducing the key-value pair feature map into the self-attention calculation module, the model can better capture the spatial and temporal dependencies in the input data, which helps the model to more accurately understand and analyze the structure and patterns in the data when processing complex data such as images, videos or time series.

[0058] Next, go to step S103;

[0059] At step S103,

[0060] Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0061] Specifically, the key-value pair feature map in each self-attention layer is compressed by a deformable convolution kernel to obtain a compressed key-value pair feature map, including:

[0062] At each position, an additional convolution branch is used to predict the offset in the area around the corresponding position;

[0063] The features at adjacent positions are weighted sampled according to the offset to obtain a compressed key-value pair feature map.

[0064] Furthermore, weighted sampling is performed on features at adjacent positions according to the offset, including:

[0065] Adjust the sampling position of the convolution kernel according to the predicted offset;

[0066] The convolution kernel after adjusting the sampling position is weighted summed with the elements at the corresponding position of the input feature map.

[0067] That is to say, in the embodiment of the present disclosure, convolution with offset allows the convolution kernel to move to a new position on the input feature map for sampling, and this new sampling position is determined by the original position and the offset; by applying the offset, the convolution operation can more accurately locate the important local area in the input feature map, thereby extracting more meaningful features.

[0068] Further, in the embodiment of the present disclosure, the weighted sampling process adopts a weighted method of deformable convolution, and the weighted method of deformable convolution includes offset learning, sampling position adjustment, weight allocation and weighted summation in the weighted process;

[0069] Specifically, the learning process of the offset is as follows: in the Deformable convolution, an offset field is first learned. This offset field is usually learned through an additional convolutional layer, and its output has the same spatial resolution as the input feature map. Each offset represents the offset of the sampling point at the corresponding position relative to the standard convolution kernel position.

[0070] The process of adjusting the sampling position is as follows: according to the learned offset, the Deformable convolution adjusts the position of its sampling point, which allows the convolution kernel to sample irregularly on the input feature map, thereby better capturing changes in shape and size.

[0071] The process of adjusting the sampling position is as follows: after determining the sampling position, the Deformable convolution assigns a weight to each sampling position. These weights usually come from the weight of the convolution kernel and are multiplied by the elements at the corresponding position on the input feature map.

[0072] The weighted summation process is as follows: Deformable convolution sums the weighted values ​​of all sampling positions to calculate the element value of the corresponding position on the output feature map. This process is similar to the weighted summation of standard convolution, but because the sampling position and weight distribution are variable, Deformable convolution can adapt to the deformation of the input features more flexibly.

[0073] Next, go to step S104;

[0074] At step S104,

[0075] Using the compressed key-value pair feature graph to train the diffusion transformer model to be generated, to obtain the diffusion transformer model;

[0076] Specifically, during the training process of the diffusion transformer model, the stochastic gradient descent SGD algorithm is used to perform joint end-to-end training on the diffusion transformer model, and the training goal is to minimize the task loss; it should be noted that in the joint end-to-end training, all parameters of the diffusion transformer model, including the front-end feature extractor and the middle conversion layer, as well as the back-end output layer, will be updated simultaneously, and the training process is usually based on a global loss function, which takes into account the performance of all relevant tasks or objectives. In this way, the diffusion transformer model can learn the optimal mapping relationship from the original input to the final output.

[0077] The reasoning process of the diffusion transformer model includes: generating corresponding keys and values ​​for word units through the self-attention layer in the diffusion transformer model to be generated; reordering the keys and values ​​into the dimensions of image and video features to obtain a key-value pair feature map; compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map; inputting the compressed key-value pair feature map into a subsequent module of the diffusion transformer model (for example, a self-attention calculation module), thereby obtaining an image or video corresponding to the word unit.

[0078] In the disclosed embodiment, the diffusion transformer model is based on the diffusion transformer model DiT (Diffusion Transformer) key-value pair feature map, which is often used to calculate the attention score and perform weighted aggregation based on it.

[0079] Specifically, the reasoning process after obtaining the key-value pair feature map usually includes: query generation, attention score calculation, score scaling, attention weight calculation, weighted value aggregation, output processing and iterative processing.

[0080] For example, during query generation, the query is used to match the key to determine which values ​​should be paid attention to;

[0081] In the process of attention score calculation, for each query, the dot product or other similarity measure with all keys is calculated to generate a score matrix, in which each element represents the matching degree of the query with the corresponding key;

[0082] During fractional scaling, in some embodiments, especially in Transformer, the attention scores are divided by a scaling factor, which is usually the square root of the dimension of the key, to avoid the vanishing gradient problem.

[0083] During the attention weight calculation process, the attention scores are converted into probability distributions, namely attention weights, by applying the softmax function; this ensures that the sum of all weights is 1 and that higher scores correspond to higher weights.

[0084] During weighted value aggregation, the values ​​are weighted summed using the attention weights; this produces an output vector that is a weighted combination of the values, where the weights are determined by the degree of match between the query and the key.

[0085] During the output processing, the weighted aggregated value may be directly used as the output of the diffusion transformer model, or may be further transformed linearly and / or subjected to nonlinear activation functions. In some embodiments, there may also be additional fully connected layers, normalized layers, or residual connections.

[0086] During the iterative process, if the diffusion transformer model contains multiple attention layers, the above process will be repeated in each layer; the output of each layer will serve as the input of the next layer, and as the network goes deeper, more complex patterns and relationships can be captured.

[0087] The following is a pseudo code example of a diffusion transformer model according to an embodiment of the present application:

[0088] #Image DiT self-attention layer

[0089] keys = keys.reshape(b,h,w,c)

[0090] values=values.reshape(b,h,w,c)

[0091] #VideoDiT self-attention layer

[0092] keys=keys.reshape(b,h,w,t,c)

[0093] values=values.reshape(b,h,w,t,c)

[0094] #Insert deformable convolution kernel

[0095] offsets=deform_conv_offsets(keys)

[0096] compressed_keys=deform_conv_sampling(keys,offsets)

[0097] compressed_values=deform_conv_sampling(values,offsets)

[0098] # Input the compressed key-value pairs to the subsequent self-attention calculation

[0099] attn_weights=attention(q,compressed_keys)

[0100] attn_outputs=attn_weights*compressed_values

[0101] Figure 2 A system 200 for constructing a diffusion transformer model according to an embodiment of the present disclosure is shown, comprising:

[0102] The initialization unit 201 is configured as follows:

[0103] Used to generate corresponding keys and values ​​by passing the word unit through the self-attention layer in the diffusion transformer model to be generated;

[0104] The processing unit 202 is configured to:

[0105] Used to reorder the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0106] The compression unit 203 is configured as follows:

[0107] Used to compress the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0108] The construction unit 204 is configured to:

[0109] Used to train the diffusion transformer model to be generated by using the compressed key-value pair feature map to obtain the diffusion transformer model.

[0110] A third embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0111] at least one processor; and,

[0112] a memory communicatively connected to the at least one processor; wherein,

[0113] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a diffusion transformer model according to any of the aforementioned embodiments.

[0114] The fourth embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method or reasoning method for constructing a diffusion transformer model described in any of the aforementioned embodiments.

[0115] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method or reasoning method for constructing a diffusion transformer model of any of the aforementioned embodiments.

[0116] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the method for constructing a diffusion transformer model or the reasoning method of any of the aforementioned embodiments.

[0117] Figure 3 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown, which may include more or fewer devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple devices. In some embodiments, it can be implemented using cloud or distributed devices.

[0118] like Figure 3 As shown, the device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) 1002 or the program and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0119] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0120] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 3 Only some components are schematically shown in the figure, which does not mean that the device 1000 only includes Figure 3 Components shown.

[0121] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0122] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method or reasoning method for constructing a diffusion transformer model described in the embodiment is implemented.

[0123] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0124] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the method or reasoning method for constructing a diffusion transformer model described in the embodiment.

[0125] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0126] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0127] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0128] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0129] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

[0130] In addition, the method, system, device, and medium according to the present invention can also be implemented in the following manner:

[0131] (1) A method for constructing a diffusion transformer model, characterized in that it includes the following steps:

[0132] Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value;

[0133] Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0134] Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0135] The diffusion transformer model to be generated is trained using the compressed key-value pair feature map to obtain the diffusion transformer model.

[0136] (2) The method for constructing a diffusion transformer model according to (1) is characterized in that if the diffusion transformer model is used to generate an image, the offset includes an x ​​offset and a y offset, and the image video feature includes an image two-dimensional feature.

[0137] (3) The method for constructing a diffusion transformer model according to (1) is characterized in that if the diffusion transformer model is used to generate a video, the offset includes an x ​​offset, a y offset and a t offset, and the image video feature includes a video three-dimensional feature.

[0138] (4) The method for constructing a diffusion transformer model according to (1), characterized in that the key-value pair feature map in each self-attention layer is compressed by a deformable convolution kernel to obtain a compressed key-value pair feature map, including:

[0139] At each position, an additional convolution branch is used to predict the offset in the area around the corresponding position;

[0140] The features at adjacent positions are weighted sampled according to the offset to obtain a compressed key-value pair feature map.

[0141] (5) The method for constructing a diffusion transformer model according to (4), characterized in that weighted sampling of features at adjacent positions is performed according to the offset, comprising:

[0142] Adjust the sampling position of the convolution kernel according to the predicted offset;

[0143] The convolution kernel after adjusting the sampling position is weighted summed with the elements at the corresponding position of the input feature map.

[0144] (6) The method for constructing a diffusion transformer model according to (4) is characterized in that the weighted sampling process adopts a weighted method of deformable convolution, and the weighted method of deformable convolution includes offset learning, sampling position adjustment, weight allocation and weighted summation in the weighted process.

[0145] (7) A method for inferring a diffusion transformer model constructed by using the method for constructing a diffusion transformer model according to any one of (1) to (6), characterized by comprising:

[0146] Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value;

[0147] Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0148] Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0149] The compressed key-value pair feature map is input into the subsequent module of the diffusion transformer model to obtain an image or video corresponding to the word unit.

[0150] (8) A system for constructing a diffusion transformer model, comprising:

[0151] Initialization unit, configured as:

[0152] Used to generate corresponding keys and values ​​by passing the word unit through the self-attention layer in the diffusion transformer model to be generated;

[0153] The processing unit is configured as follows:

[0154] Used to reorder the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map;

[0155] Compression units are configured as follows:

[0156] Used to compress the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map;

[0157] The building blocks are configured as follows:

[0158] Used to train the diffusion transformer model to be generated by using the compressed key-value pair feature map to obtain the diffusion transformer model.

[0159] (9) A system for constructing a diffusion transformer model according to (8), characterized in that:

[0160] If the diffusion transformer model is used to generate an image, the offset includes an x ​​offset and a y offset, and the image video feature includes an image two-dimensional feature.

[0161] (10) A system for constructing a diffusion transformer model according to (8), characterized in that:

[0162] If the diffusion transformer model is used to generate video, the offset includes an x ​​offset, a y offset, and a t offset, and the image video feature includes a video three-dimensional feature.

[0163] (11) A system for constructing a diffusion transformer model according to (8), characterized in that:

[0164] The compression unit is further configured to:

[0165] Used to predict the offset in the area around the corresponding position through an additional convolution branch at each position;

[0166] The features at adjacent positions are weighted sampled according to the offset to obtain a compressed key-value pair feature map.

[0167] (12) A system for constructing a diffusion transformer model according to (8), characterized in that:

[0168] Used to adjust the sampling position of the convolution kernel according to the predicted offset;

[0169] The convolution kernel after adjusting the sampling position is weighted summed with the elements at the corresponding position of the input feature map.

[0170] (13) A system for constructing a diffusion transformer model according to (8), characterized in that:

[0171] The process for weighted sampling adopts a weighted method of deformable convolution, which includes learning of offset, adjustment of sampling position, weight allocation and weighted summation in the weighted process.

[0172] (14) An electronic device, characterized in that the electronic device comprises:

[0173] at least one processor; and,

[0174] a memory communicatively connected to the at least one processor; wherein,

[0175] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of constructing a diffusion transformer model described in any one of (1) to (6) or the reasoning method described in (7).

[0176] (15) A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method of constructing a diffusion transformer model described in any one of (1) to (6) or the reasoning method described in (7).

Claims

1. A method for constructing a diffusion transformer model, characterized in that: The following steps are involved: Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value; Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map; Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map; The method of compressing the key-value pair feature map in each self-attention layer by using a deformable convolution kernel to obtain a compressed key-value pair feature map includes: At each position, an additional convolution branch is used to predict the offset in the area around the corresponding position; Performing weighted sampling on features at adjacent positions according to the offset to obtain a compressed key-value pair feature map; The diffusion transformer model to be generated is trained using the compressed key-value pair feature map to obtain the diffusion transformer model.

2. The method for constructing a diffusion transformer model according to claim 1, characterized in that: If the diffusion transformer model is used to generate an image, the offset includes an x ​​offset and a y offset, and the image video feature includes an image two-dimensional feature.

3. The method for constructing a diffusion transformer model according to claim 1, characterized in that: If the diffusion transformer model is used to generate video, the offset includes an x ​​offset, a y offset, and a t offset, and the image video feature includes a video three-dimensional feature.

4. The method for constructing a diffusion transformer model according to claim 1, characterized in that: The features at adjacent positions are weighted sampled according to the offset, including: Adjust the sampling position of the convolution kernel according to the predicted offset; The convolution kernel after adjusting the sampling position is weighted summed with the elements at the corresponding position of the input feature map.

5. The method for constructing a diffusion transformer model according to claim 1, characterized in that: The weighted sampling process adopts a weighted method of deformable convolution, and the weighted method of deformable convolution includes offset learning, sampling position adjustment, weight allocation and weighted summation in the weighted process.

6. A method for inferring a diffusion transformer model constructed by using the method for constructing a diffusion transformer model according to any one of claims 1 to 5, characterized in that: include: Pass the word unit through the self-attention layer in the diffusion transformer model to be generated to generate the corresponding key and value; Rearrange the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map; Compressing the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map; The compressed key-value pair feature map is input into the subsequent module of the diffusion transformer model to obtain an image or video corresponding to the word unit.

7. A system for constructing a diffusion transformer model, characterized in that: include: Initialization unit, configured as: Used to generate corresponding keys and values ​​by passing the word unit through the self-attention layer in the diffusion transformer model to be generated; The processing unit is configured as follows: Used to reorder the keys and the values ​​into dimensions of image and video features to obtain a key-value pair feature map; Compression units are configured as follows: Used to compress the key-value pair feature map in each self-attention layer through a deformable convolution kernel to obtain a compressed key-value pair feature map, including: predicting an offset in the area around the corresponding position through an additional convolution branch at each position, and performing weighted sampling on features of adjacent positions according to the offset to obtain a compressed key-value pair feature map; The building blocks are configured as: Used to train the diffusion transformer model to be generated by using the compressed key-value pair feature map to obtain the diffusion transformer model.

8. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a diffusion transformer model as described in any one of claims 1 to 5 or the reasoning method as described in claim 6.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable the computer to execute the method for constructing a diffusion transformer model as described in any one of claims 1 to 5 or the reasoning method as described in claim 6.

Images