A reasoning method, system, device and storage medium for a diffusion transformer model based on a window attention layer

By introducing window attention layer and residual reuse mechanism into the DiT model, the problem of low computing efficiency of DiT model is solved, and efficient computing is achieved in resource-constrained environments, suitable for high-resolution image and video generation.

CN118536554BActive Publication Date: 2024-11-29BEIJING WUWEN CORE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410699167.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-31
Publication Date
2024-11-29
Estimated Expiration
2044-05-31

AI Technical Summary

Technical Problem

The DiT model is incomputed in processing high-resolution images and videos, resulting in huge consumption of computing resources and is difficult to actually apply on terminal devices. The existing LocalAttention method cannot effectively utilize Diffusion time series information, and DeepCache can only be applied to U-Net On the model, it cannot be applied to the DiT model.

Method used

Using a diffusion converter model based on the window attention layer, the residual threshold is set between the full attention layer and the window attention layer through the WA-RS method, and the full attention layer or window attention layer is selected according to the actual output residual. Processing is performed and the actual output residual is reused in subsequent iterations to reduce the amount of calculation.

Benefits of technology

It effectively reduces the computational complexity and resource consumption of the DiT model, maintains model performance, is suitable for resource-constrained environments, and significantly improves computing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118536554B_ABST
    Figure CN118536554B_ABST
Patent Text Reader

Abstract

The present invention provides an inference method, system, device and storage medium of a diffusion transformer model based on a window attention layer, comprising the following steps: inputting a potential representation of an image into a diffusion transformer model for iteration; the diffusion transformer model obtains the actual output residual of the potential representation of the image between a full attention layer and a window attention layer based on a WA‑RS method, and presets a residual threshold; during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image; after the iteration of the diffusion transformer model is completed, the target image or video of the potential representation of the image is output; the present application effectively reduces the amount of calculation in the DiT model by introducing a fixed-size window attention layer and caching and reuse of residuals through the WA‑RS method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and specifically relates to an inference method, system, device and storage medium of a diffusion transformer model based on a window attention layer. Background Art

[0002] In recent years, inspired by the great success of the Transformer model in natural language processing tasks, the Diffusion Model has received extensive attention and application in the field of computer vision. Among them, the DiT (Diffusion Transformer) model introduces the self-attention mechanism into the diffusion process, showing excellent large-resolution image generation and video generation capabilities. However, the DiT model generates a large number of key-value pairs when processing high-resolution images and videos, resulting in the need to consume huge computing resources to calculate attention. This not only increases the computational complexity of the model, but also greatly limits its practical application on terminal devices. Therefore, effectively accelerating attention calculation has become a key challenge to improve the efficiency of the DiT model.

[0003] Among the existing related technologies, the core idea of ​​Deepcache is to reuse the high-level features in the U-Net model and skip the calculation of the intermediate layer to speed up the denoising process. In this way, Deepcache can effectively reduce the amount of calculation while maintaining the accuracy and performance of the model. Specifically, Deepcache introduces a cache mechanism in the U-Net model to store the high-level features calculated previously. In subsequent time steps, if similar features are encountered, Deepcache can directly extract the corresponding high-level features from the cache without recalculating the intermediate layer. In this way, the model can skip some unnecessary calculation steps, thereby improving the efficiency of the denoising process. This technical solution of Deepcache can not only significantly improve the running speed of the model, but also reduce the computing resource consumption of the model, which is of great practical significance for application scenarios that need to process large amounts of data and have high real-time requirements.

[0004] The local attention mode allows each token to only focus on neighboring tokens within a fixed window size. This method aims to reduce the computational burden when processing long sequences. The concept of local windowed attention was originally proposed by Longformer, which demonstrated an attention mechanism that scales linearly with the length of the sequence. Bigbird further extended this concept by combining window attention, random attention, and global attention mechanisms to preserve long-distance dependencies and reduce computational costs. In the field of computer vision, SwinTransformer adopts a similar strategy by limiting attention calculations to non-overlapping local windows and using shifted windows between different layers to effectively capture global context. Models such as TwinsTransformer and FasterViT use window-based attention to improve computational efficiency and use different module designs such as global subsampling attention and hierarchical attention to effectively utilize global context.

[0005] However, the DeepCache solution can only be used on UNet and cannot be applied to the DiT model. The existing LocalAttention does not utilize the information on the Diffusion time series, and using Local Attention will significantly reduce the quality. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention provides a method, system, device and storage medium for a diffusion transformer model based on a window attention layer, so as to solve the technical problem of low computational efficiency of the DiT model in the related prior art.

[0007] In a first aspect, an embodiment of the present invention provides an inference method of a diffusion transformer model based on a window attention layer, comprising the following steps:

[0008] The image latent representation is fed into the diffusion transformer model for iteration;

[0009] The diffusion transformer model obtains the actual output residual of the potential representation of the image between the full attention layer and the window attention layer based on the WA-RS method, and presets the residual threshold;

[0010] During the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image;

[0011] After the iteration of the diffusion transformer model is completed, the target image or video potentially represented by the image is output.

[0012] Furthermore, if a windowed attention layer is used to process the potential representation of the image, the actual output residual is reused in subsequent iterative optimization of the diffusion transformer model.

[0013] Further, if in a subsequent iterative optimization of the diffusion transformer model, a full attention layer is present to process the potential representation of the image, then the actual output residual is cached;

[0014] The cached actual output residual is used in the iteration of the window attention layer during the subsequent iteration of the diffusion transformer model, and the cached actual output residual generated in the first iteration is reused in the iteration of the window attention layer this time.

[0015] Furthermore, the actual output residual is:

[0016] Rr=Or-Wk;

[0017] Among them, Or is the output of the full attention layer, and Wk is the output of the window attention layer.

[0018] When the actual output residual is reused:

[0019] Ok=Wk+Rr;

[0020] Among them, Ok is the output after reusing the actual output residual.

[0021] Furthermore, the WA-RS method is an architecture based on a window attention mechanism and residual connections.

[0022] Furthermore, the architecture based on the window attention mechanism and the residual connection, when used, includes the following steps:

[0023] The input features are processed through the window attention layer to capture local area information;

[0024] The captured local area information is added to the actual output residual to obtain the output of this iteration.

[0025] Furthermore, the architecture based on window attention mechanism and residual connection is set in all SwinTransformer blocks in the diffusion transformer model.

[0026] In an embodiment of the present invention, in a second aspect, an embodiment of the present invention provides an inference system of a diffusion transformer model based on a window attention layer, comprising:

[0027] Input unit, configured as:

[0028] Used to input the image potential representation into the diffusion transformer model for iteration;

[0029] The preprocessing unit is configured as follows:

[0030] The diffusion transformer model is used to obtain the actual output residual between the full attention layer and the window attention layer of the potential representation of the image based on the WA-RS method, and a residual threshold is preset;

[0031] The selection unit is configured as:

[0032] Used for, during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, using the full attention layer to process the potential representation of the image, otherwise, using the window attention layer to process the potential representation of the image;

[0033] Output unit, configured as:

[0034] After the diffusion transformer model iteration is completed, the target image or video potentially represented by the image is output.

[0035] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0036] at least one processor; and,

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned reasoning method of the diffusion transformer model based on the window attention layer.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned inference method of the diffusion transformer model based on the window attention layer.

[0040] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document.

[0041] Compared with the prior art, the present invention has the following beneficial technical effects:

[0042] The present invention provides a method, system, device and storage medium for a diffusion transformer model based on a window attention layer, comprising the following steps: inputting a potential representation of an image into a diffusion transformer model for iteration; the diffusion transformer model obtains the actual output residual of the potential representation of the image between a full attention layer and a window attention layer based on a WA-RS method, and presets a residual threshold; during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image; after the iteration of the diffusion transformer model is completed, the target image or video of the potential representation of the image is output; the present application effectively reduces the amount of calculation in the DiT model by introducing a fixed-size window attention layer and caching and reuse of residuals through the WA-RS method, while maintaining the model performance. Through this method, the computational complexity of the model can be significantly reduced without sacrificing too much performance, making it more suitable for resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0044] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0045] Figure 1 A flow chart of an inference method of a diffusion transformer model based on a window attention layer according to an embodiment of the present disclosure is shown;

[0046] Figure 2 A flowchart showing the caching and reuse of the window attention layer and residuals of an embodiment of the present disclosure is shown;

[0047] Figure 3 A schematic diagram of the system structure of an inference method of a diffusion transformer model based on a window attention layer according to an embodiment of the present disclosure is shown;

[0048] Figure 4 An exemplary structural diagram of a device capable of implementing a method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0050] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0051] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0052] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0053] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0054] Figure 1 A method 100 of a diffusion transformer model based on a window attention layer according to an embodiment of the present disclosure is shown. The embodiment of the present disclosure includes the following steps:

[0055] At step S101,

[0056] The image latent representation is fed into the diffusion transformer model for iteration;

[0057] In the disclosed embodiment, the image potential representation refers to the low-dimensional vectors or features learned from the original image data, which can capture the key information, intrinsic structure and characteristics of the image. Specifically, the diffusion transformer model performs step-by-step noise addition based on the original image data. The step-by-step noise addition is to gradually add Gaussian noise to the image over multiple time steps, so that it gradually "diffuses" to a pure noise state. This process simulates the diffusion process of non-equilibrium thermodynamics. In the middle stage of the diffusion process, the image is no longer completely clear or completely noisy, but is in a state between the two. These intermediate states can be regarded as the image potential representation described in this embodiment.

[0058] Next, go to step S102;

[0059] At step S102, the diffusion transformer model obtains the actual output residual of the potential representation of the image between the full attention layer and the window attention layer based on the WA-RS method, and presets a residual threshold;

[0060] It should be noted that the attention range of the full attention layer covers the entire input sequence, which means that when calculating the output of any given position, the full attention layer will consider all positions in the input sequence. This global attention mechanism enables the model to capture long-distance dependencies, but it also brings higher computational costs. The window attention layer is different from the full attention layer. The window attention layer only focuses on a fixed-size window in the input sequence, or called a local area. Within this window, the model calculates the attention weights and generates outputs. The main advantage of the window attention layer is its lower computational cost because the entire input sequence does not need to be considered. However, this local attention mechanism may not be able to effectively capture long-distance dependencies. In the self-attention mechanism of the diffusion transformer model of this embodiment, the computational complexity is reduced by limiting each token to only focus on other tokens in its local window, rather than the entire sequence. It should be noted that the token is a potential representation of the image.

[0061] It should be further explained that the residual threshold is for the same set of data to be processed, and the reversibility of the generated results should not be destroyed. That is to say, when the actual output residual back-weighted full attention layer is used, it can ensure the partial or most of the position information of the long sequence; for those skilled in the art, the appropriate residual threshold should be selected according to the task requirements and the characteristics of the data set.

[0062] In the disclosed embodiment, the WA-RS method is an architecture based on a window attention mechanism and a residual connection.

[0063] Furthermore, the architecture based on the window attention mechanism and the residual connection, when used, includes the following steps:

[0064] The input features are processed through the window attention layer to capture local area information;

[0065] The captured local area information is added to the actual output residual to obtain the output of this iteration.

[0066] Specifically, Figure 2 A flowchart of the window attention layer and residual caching and reuse in an embodiment of the present disclosure is shown, as shown in FIG. Figure 2 As shown, the diffusion transformer model needs to perform multiple iterations in operation, and each iteration performs a model inference. Step t in the figure represents the t-th iterative optimization process. It can be seen from the figure that the diffusion transformer model obtains the actual output residual between the full attention layer and the window attention layer of the potential representation of the image based on the WA-RS method;

[0067] Among them, in step t, the actual output residual is R t , the output of the full attention layer is The output of its window attention layer is

[0068] In the subsequent step t+1, the window attention layer is used to process the data to obtain The actual output is

[0069] In the subsequent step t+2, the window attention layer is used to process the data to obtain The actual output is

[0070] Furthermore, the architecture based on window attention mechanism and residual connection is set in all SwinTransformer blocks in the diffusion transformer model.

[0071] It should be noted that the SwinTransformer block uses two types of window-based multi-head self-attention (Window-basedMulti-headSelf-Attention, also known as W-MSA and SW-MSA), where the W-MSA represents a window-based multi-head self-attention using a regular window segmentation configuration, which divides the image into multiple windows with a fixed-size window for self-attention calculation; the SW-MSA represents a window-based multi-head self-attention using a shifted window segmentation configuration, which improves cross-window information interaction through a sliding window mechanism. This window-based and shifted window self-attention mechanism can improve computational efficiency while retaining sufficient global context information.

[0072] Next, go to step S103;

[0073] At step S103, during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image;

[0074] In this embodiment, if a window attention layer is used to process the image potential representation, the actual output residual is reused in subsequent iterative optimization of the diffusion transformer model.

[0075] Specifically, if in a subsequent iterative optimization of the diffusion transformer model, there is a full attention layer to process the potential representation of the image, then caching the actual output residual;

[0076] The cached actual output residual is used in the iteration of the window attention layer during the subsequent iteration of the diffusion transformer model, and the cached actual output residual generated in the first iteration is reused in the iteration of the window attention layer this time.

[0077] More specifically, the actual output residual is:

[0078] Rr=Or-Wk;

[0079] Among them, Or is the output of the full attention layer, and Wk is the output of the window attention layer.

[0080] When the actual output residual is reused:

[0081] Ok=Wk+Rr;

[0082] Among them, Ok is the output after reusing the actual output residual.

[0083] Next, go to step S104;

[0084] At step S104, after the iteration of the diffusion transformer model is completed, the target image or video potentially represented by the image is output.

[0085] Figure 3 A system 200 of a diffusion transformer model based on a window attention layer according to an embodiment of the present disclosure is shown, comprising:

[0086] The input unit 201 is configured as follows:

[0087] Used to input the image potential representation into the diffusion transformer model for iteration;

[0088] The pre-processing unit 202 is configured to:

[0089] The diffusion transformer model is used to obtain the actual output residual between the full attention layer and the window attention layer of the potential representation of the image based on the WA-RS method, and a residual threshold is preset;

[0090] The selection unit 203 is configured to:

[0091] Used for, during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, using the full attention layer to process the potential representation of the image, otherwise, using the window attention layer to process the potential representation of the image;

[0092] The output unit 204 is configured as follows:

[0093] After the diffusion transformer model iteration is completed, the target image or video potentially represented by the image is output.

[0094] A third embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0095] at least one processor; and,

[0096] a memory communicatively connected to the at least one processor; wherein,

[0097] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the diffusion transformer model based on the window attention layer of any of the aforementioned embodiments.

[0098] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the method of the diffusion transformer model based on the window attention layer as described in any of the aforementioned embodiments.

[0099] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method of the diffusion transformer model based on the window attention layer of any of the aforementioned embodiments.

[0100] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the method of the diffusion transformer model based on the window attention layer of any of the aforementioned embodiments.

[0101] Figure 4 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown, which may include more or fewer devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple devices. In some embodiments, it can be implemented using cloud or distributed devices.

[0102] like Figure 4 As shown, the device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) 1002 or the program and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM1003, various programs and data required for the operation of device 1000 are also stored. Processor 1001, ROM1002 and RAM1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0103] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0104] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 4 Only some components are schematically shown in the figure, which does not mean that the device 1000 only includes Figure 4 Components shown.

[0105] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0106] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method of the diffusion transformer model based on the window attention layer described in the embodiment is implemented.

[0107] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0108] Although not shown, an embodiment of the present invention also provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the method of the window attention layer-based diffusion transformer model described in the embodiment.

[0109] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0110] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0111] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0112] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0113] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

[0114] In addition, the method, system, device, and medium according to the present invention can also be implemented in the following manner:

[0115] (1) A method for a diffusion transformer model based on a window attention layer, characterized in that it includes the following steps:

[0116] The image latent representation is fed into the diffusion transformer model for iteration;

[0117] The diffusion transformer model obtains the actual output residual of the potential representation of the image between the full attention layer and the window attention layer based on the WA-RS method, and presets the residual threshold;

[0118] During the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image;

[0119] After the iteration of the diffusion transformer model is completed, the target image or video potentially represented by the image is output.

[0120] (2) The method of the diffusion transformer model based on the window attention layer according to (1) is characterized in that if the window attention layer is used to process the potential representation of the image, the actual output residual is reused in the subsequent iterative optimization of the diffusion transformer model.

[0121] (3) The method of the diffusion transformer model based on the window attention layer according to (2), characterized in that if in the subsequent iterative optimization of the diffusion transformer model, there is a full attention layer to process the potential representation of the image, then the actual output residual is cached;

[0122] The cached actual output residual is used in the iteration of the window attention layer during the subsequent iteration of the diffusion transformer model, and the cached actual output residual generated in the first iteration is reused in the iteration of the window attention layer this time.

[0123] (4) According to the method of the diffusion transformer model based on the window attention layer described in (2), the method of the diffusion transformer model based on the window attention layer is characterized in that the actual output residual is:

[0124] Rr=Or-Wk;

[0125] Among them, Or is the output of the full attention layer, and Wk is the output of the window attention layer.

[0126] When the actual output residual is reused:

[0127] Ok=Wk+Rr;

[0128] Among them, Ok is the output after reusing the actual output residual.

[0129] (5) According to the method of the diffuse transformer model based on the window attention layer described in (1), the method of the diffuse transformer model based on the window attention layer is characterized in that the WA-RS method is an architecture based on the window attention mechanism and residual connection.

[0130] (6) The method of the diffuse transformer model based on the window attention layer according to (5) is characterized in that the architecture based on the window attention mechanism and the residual connection, when used, comprises the following steps:

[0131] The input features are processed through the window attention layer to capture local area information;

[0132] The captured local area information is added to the actual output residual to obtain the output of this iteration.

[0133] (7) The method of the diffusion transformer model based on the window attention layer according to (5) is characterized in that the architecture based on the window attention mechanism and residual connection is set in all SwinTransformer blocks in the diffusion transformer model.

[0134] (8) A system based on a diffusion transformer model of a window attention layer, characterized by comprising:

[0135] Input unit, configured as:

[0136] Used to input the image potential representation into the diffusion transformer model for iteration;

[0137] The preprocessing unit is configured as follows:

[0138] The diffusion transformer model is used to obtain the actual output residual between the full attention layer and the window attention layer of the potential representation of the image based on the WA-RS method, and a residual threshold is preset;

[0139] The selection unit is configured as:

[0140] Used for, during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, using the full attention layer to process the potential representation of the image, otherwise, using the window attention layer to process the potential representation of the image;

[0141] Output unit, configured as:

[0142] After the diffusion transformer model iteration is completed, the target image or video potentially represented by the image is output.

[0143] (9) The system according to (8), characterized in that

[0144] The pre-processing unit is further configured to:

[0145] Used to process input features through the window attention layer to capture local area information;

[0146] The captured local area information is added to the actual output residual to obtain the output of this iteration.

[0147] (10) The system according to (8), characterized in that

[0148] The selection unit is further configured to:

[0149] If a window attention layer is used to process the potential representation of the image, the actual output residual is reused in subsequent iterative optimization of the diffusion transformer model.

[0150] (11) The system according to (8), characterized in that

[0151] The selection unit is further configured to:

[0152] A method for a diffuse transformer model based on a windowed attention layer, characterized in that if in a subsequent iterative optimization of the diffuse transformer model, a full attention layer is present to process a potential representation of the image, then the actual output residual is cached;

[0153] The cached actual output residual is used in the iteration of the window attention layer during the subsequent iteration of the diffusion transformer model, and the cached actual output residual generated in the first iteration is reused in the iteration of the window attention layer this time.

[0154] (12) An electronic device, characterized in that the electronic device comprises:

[0155] at least one processor; and,

[0156] a memory communicatively connected to the at least one processor; wherein,

[0157] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the diffusion transformer model based on the window attention layer described in any one of (1) to (7).

[0158] (13) A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method of the diffusion transformer model based on the window attention layer described in any one of (1) to (7).

Claims

1. An inference method for a diffusion transformer model based on a window attention layer, characterized in that: The following steps are involved: The image latent representation is fed into the diffusion transformer model for iteration; The diffusion transformer model obtains the actual output residual of the potential representation of the image between the full attention layer and the window attention layer based on the window attention and residual sharing WA-RS method, and presets the residual threshold, wherein the window attention and residual sharing WA-RS method is an architecture based on the window attention mechanism and residual connection. When the architecture based on the window attention mechanism and residual connection is used, the input feature is processed by the window attention layer to capture local area information, and the captured local area information is added to the actual output residual to obtain the output of this iteration; wherein During the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, the full attention layer is used to process the potential representation of the image, otherwise, the window attention layer is used to process the potential representation of the image; After the iteration of the diffusion transformer model is completed, the target image or video potentially represented by the image is output.

2. The inference method of the diffusion transformer model based on the window attention layer according to claim 1 is characterized in that: If a windowed attention layer is used to process the image latent representation, the actual output residual is reused in subsequent iterative optimizations of the diffusion transformer model.

3. The inference method of the diffusion transformer model based on the window attention layer according to claim 2 is characterized in that: If, in a subsequent iterative optimization of the diffusion transformer model, a full attention layer is present to process the latent representation of the image, then caching the actual output residual; The cached actual output residual is used in the iteration of the window attention layer during the subsequent iteration of the diffusion transformer model, and the cached actual output residual generated in the first iteration is reused in the iteration of the window attention layer this time.

4. The inference method of the diffusion transformer model based on the window attention layer according to claim 2 is characterized in that: The actual output residual is: Rr=Or-Wk; Among them, Or is the output of the full attention layer, and Wk is the output of the window attention layer; When the actual output residual is reused: Ok=Wk+Rr; Among them, Ok is the output after reusing the actual output residual.

5. The inference method of the diffusion transformer model based on the window attention layer according to claim 1 is characterized in that: The architecture based on window attention mechanism and residual connection is set in all SwinTransformer blocks in the diffusion transformer model.

6. An inference system based on a diffusion transformer model with a window attention layer, characterized in that: include: Input unit, configured as: Used to input the image potential representation into the diffusion transformer model for iteration; The preprocessing unit is configured as follows: The diffusion transformer model is used to obtain the actual output residual of the potential representation of the image between the full attention layer and the window attention layer based on the window attention and residual sharing WA-RS method, and preset the residual threshold, wherein the window attention and residual sharing WA-RS method is an architecture based on the window attention mechanism and residual connection. When the architecture based on the window attention mechanism and residual connection is used, the input feature is processed by the window attention layer to capture local area information, and the captured local area information is added to the actual output residual to obtain the output of this iteration; The selection unit is configured as: Used for, during the iteration of the diffusion transformer model, if the actual output residual is greater than the residual threshold, using the full attention layer to process the potential representation of the image, otherwise, using the window attention layer to process the potential representation of the image; Output unit, configured as: After the diffusion transformer model iteration is completed, the target image or video potentially represented by the image is output.

7. An electronic device, characterized in that: The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the inference method of the diffusion transformer model based on the window attention layer as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable the computer to execute the inference method of the diffusion transformer model based on the window attention layer as described in any one of claims 1 to 5.