Time step adjustment method and device of diffusion model, electronic equipment and storage medium

Abstract

The invention discloses a time step adjustment method and device of a diffusion model, electronic equipment and a storage medium. The method comprises the following steps: acquiring a video resolution of a target video and a video frame number of the target video; determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video; an initial time step of a diffusion model is obtained, non-linear adjustment is carried out on the initial time step of the diffusion model based on the time step offset parameter, a target time step of the diffusion model is obtained, and the target time step is used for controlling the diffusion model to carry out noise injection and noise removal on the target video. According to the technical scheme, the time step offset parameter is dynamically calculated according to the video resolution and the video frame number of the target video, and then the target time step adaptive to the target video characteristics is generated according to the time step offset parameter, so that when the target time step is used for executing the video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a time step adjustment method, device, electronic device and storage medium for a diffusion model. Background Art

[0002] In the video generation task, the time step of the diffusion model can directly affect the video generation quality and noise processing efficiency. Specifically, the time step is used to guide the noise injection and denoising process. It defines the noise intensity and signal ratio of the diffusion model in each stage and is the core parameter of the video generation process.

[0003] In the prior art, the design of time steps is usually based on the following two methods:

[0004] This method uses a fixed linear time step sequence to perform denoising and noise removal according to a linearly uniformly changing time step. This method is simple and easy to use, but it cannot be adaptively adjusted according to the characteristics of the input video, which may lead to low generation efficiency or unstable quality.

[0005] Taking predefined distribution sampling, this method uses Gaussian distribution or other fixed distribution to generate time steps, such as sampling time steps by random noise during the model training phase to enhance the generalization ability of the model. However, this method may cause the time step distribution to not match the video characteristics under diverse input conditions.

[0006] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: the prior art solution has the problem of low video generation quality and efficiency. Summary of the invention

[0007] The present invention provides a time step adjustment method, device, electronic device and storage medium of a diffusion model to improve the quality and efficiency of video generation.

[0008] According to one aspect of the present invention, a time step adjustment method for a diffusion model is provided, comprising:

[0009] Get the video resolution and the number of video frames of the target video;

[0010] Determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video;

[0011] An initial time step of a diffusion model is obtained, and based on the time step offset parameter, a nonlinear adjustment is performed on the initial time step of the diffusion model to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0012] According to another aspect of the present invention, there is provided a time step adjustment device for a diffusion model, comprising:

[0013] A video resolution and video frame number acquisition module is used to acquire the video resolution and video frame number of a target video;

[0014] A time step offset parameter determination module, used to determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video;

[0015] The time step nonlinear adjustment module is used to obtain the initial time step of the diffusion model, and based on the time step offset parameter, nonlinearly adjust the initial time step of the diffusion model to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor;

[0018] and a memory communicatively coupled to the at least one processor;

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the time step adjustment method of the diffusion model described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the time step adjustment method of the diffusion model described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention obtains the video resolution and the number of video frames of the target video; determines the time step offset parameter based on the video resolution and the number of video frames of the target video; obtains the initial time step of the diffusion model, and nonlinearly adjusts the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise from the target video. The above technical solution dynamically calculates the time step offset parameter according to the video resolution and the number of video frames of the target video, and then generates the target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then uses the target time step to perform video generation tasks, which can effectively improve the video generation quality and efficiency of the diffusion model.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 is a flow chart of a time step adjustment method of a diffusion model provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a time step adjustment method for a diffusion model provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a flow chart of a time step adjustment method for a diffusion model provided in accordance with Embodiment 3 of the present invention;

[0027] Figure 4 is a structural schematic diagram of a time step adjustment device for a diffusion model provided according to a fourth embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the time step adjustment method of the diffusion model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a time step adjustment method for a diffusion model provided in the first embodiment of the present invention. This embodiment is applicable to the noise sampling training and reasoning of the diffusion model in the video generation task. The method can be executed by a time step adjustment device for the diffusion model. The time step adjustment device for the diffusion model can be implemented in the form of hardware and / or software. The time step adjustment device for the diffusion model can be configured in electronic devices such as terminals and / or servers. Figure 1 As shown, the method includes:

[0033] S110 , obtaining the video resolution of the target video and the number of video frames of the target video.

[0034] In the embodiment of the present invention, the target video refers to a video to be input into the diffusion model.

[0035] Exemplarily, in response to the parameter input operation, the video resolution of the target video and the number of video frames of the target video are obtained.

[0036] S120. Determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video.

[0037] S130, obtaining an initial time step of a diffusion model, and performing nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0038] In the embodiment of the present invention, the time step offset parameter refers to an offset parameter for nonlinearly adjusting the initial time step, wherein the initial time step may be a uniformly distributed time step obtained by initialization or a randomly generated time step, which is not specifically limited herein.

[0039] It can be understood that during the training process of the diffusion model, by injecting and removing noise, the diffusion model can learn how to extract and reconstruct the real video from the noisy video.

[0040] It should be noted that by dynamically calculating the time step offset parameter according to the video resolution and the number of video frames of the target video, and then generating a target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then using the target time step to perform the video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

[0041] The technical solution of the embodiment of the present invention obtains the video resolution and the number of video frames of the target video; determines the time step offset parameter based on the video resolution and the number of video frames of the target video; obtains the initial time step of the diffusion model, and nonlinearly adjusts the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise from the target video. The above technical solution dynamically calculates the time step offset parameter according to the video resolution and the number of video frames of the target video, and then generates the target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then uses the target time step to perform video generation tasks, which can effectively improve the video generation quality and efficiency of the diffusion model.

[0042] Embodiment 2

[0043] Figure 2 A flow chart of a time step adjustment method for a diffusion model provided in Embodiment 2 of the present invention, the method of this embodiment can be combined with the various optional schemes in the time step adjustment method for the diffusion model provided in the above embodiments. The time step adjustment method for the diffusion model provided in this embodiment is further optimized. Optionally, the time step offset parameter is determined based on the video resolution of the target video and the number of video frames of the target video, including: determining the spatial dimension offset parameter corresponding to the video resolution of the target video; determining the time dimension offset parameter corresponding to the number of video frames of the target video; determining the time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the time dimension offset parameter corresponding to the number of video frames of the target video.

[0044] like Figure 2 As shown, the method includes:

[0045] S210: Obtain the video resolution of the target video and the number of video frames of the target video.

[0046] S220. Determine a spatial dimension offset parameter corresponding to the video resolution of the target video.

[0047] In the embodiment of the present invention, the spatial dimension offset parameter refers to an offset parameter calculated by video resolution.

[0048] Optionally, determining the spatial dimension offset parameters corresponding to the video resolution of the target video includes: determining the spatial dimension offset parameters corresponding to the video resolution of the target video based on a preset video resolution-offset mapping relationship; wherein the preset video resolution-offset mapping relationship includes multiple video resolutions and offset parameters corresponding to each video resolution.

[0049] Exemplarily, the range of the offset parameter is (A, B), A represents the basic offset parameter, and B represents the maximum offset parameter. Specifically, the video resolution can be mapped to (A, B), and the spatial dimension offset parameter can be matched from (A, B). It should be noted that the higher the video resolution, the closer the offset parameter is to the maximum offset parameter, so that the time step is adjusted by the offset parameter corresponding to the high video resolution, so that the time step distribution is shifted to the high signal area, where the high signal area refers to the high-resolution long video. The lower the video resolution, the closer the offset parameter is to the basic offset parameter, so that the time step distribution is more uniform by adjusting the offset parameter corresponding to the low video resolution.

[0050] It should be noted that the embodiment of the present invention can dynamically adjust the time step to focus on the high signal area, enhance the detail preservation capability, and meet the needs of high-resolution long video generation. In addition, it can also reduce the calculation of low-signal areas, improve video generation efficiency, avoid redundant calculation waste, and meet the needs of low-resolution short video generation.

[0051] S230: Determine a time dimension offset parameter corresponding to the number of video frames of the target video.

[0052] In the embodiment of the present invention, the time dimension offset parameter refers to an offset parameter calculated by the number of video frames.

[0053] Specifically, the square root of the number of video frames of the target video may be determined; and the square root of the number of video frames of the target video may be determined as the time dimension offset parameter corresponding to the number of video frames of the target video.

[0054] S240. Determine a time step offset parameter based on a spatial dimension offset parameter corresponding to the video resolution of the target video and a time dimension offset parameter corresponding to the number of video frames of the target video.

[0055] Exemplarily, the time step offset parameter is obtained by multiplying the spatial dimension offset parameter corresponding to the video resolution of the target video and the temporal dimension offset parameter corresponding to the number of video frames of the target video.

[0056] S250, obtaining an initial time step of a diffusion model, and based on the time step offset parameter, performing nonlinear adjustment on the initial time step of the diffusion model to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0057] The technical solution of the embodiment of the present invention determines the spatial dimension offset parameter corresponding to the video resolution of the target video, and then determines the temporal dimension offset parameter corresponding to the video frame number of the target video, and then determines the time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the temporal dimension offset parameter corresponding to the video frame number of the target video, thereby realizing dynamic calculation of the time step offset parameter, thereby being able to adapt to diversified video generation tasks.

[0058] Embodiment 3

[0059] Figure 3 A flowchart of a time step adjustment method for a diffusion model provided in Embodiment 3 of the present invention, the method of this embodiment can be combined with the various optional schemes in the time step adjustment method for a diffusion model provided in the above embodiments. The time step adjustment method for a diffusion model provided in this embodiment is further optimized. Optionally, the nonlinear adjustment of the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model includes: determining a first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model; determining a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model; and determining the target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

[0060] like Figure 3 As shown, the method includes:

[0061] S310: Obtain the video resolution of the target video and the number of video frames of the target video.

[0062] S320: Determine a time step offset parameter based on the video resolution of the target video and the number of video frames of the target video.

[0063] S330, obtaining an initial time step of the diffusion model.

[0064] S340: Determine a first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model.

[0065] In the embodiment of the present invention, the first nonlinear time step adjustment value is a time step adjustment value calculated by using the time step offset parameter and the initial time step of the diffusion model.

[0066] Specifically, the time step offset parameter and the initial time step of the diffusion model are multiplied to obtain a first nonlinear time step adjustment value.

[0067] S350: Determine a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model.

[0068] In the embodiment of the present invention, the second nonlinear time step adjustment value refers to another time step adjustment value calculated by using the time step offset parameter and the initial time step of the diffusion model.

[0069] Specifically, the time step offset parameter is subtracted by 1 to obtain the first time step adjustment intermediate value; the first time step adjustment intermediate value and the initial time step of the diffusion model are multiplied to obtain the second time step adjustment intermediate value; and the second time step adjustment intermediate value is added by 1 to obtain the second nonlinear time step adjustment value.

[0070] S360. Determine a target time step of a diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0071] Specifically, the first nonlinear time step adjustment value is divided by the second nonlinear time step adjustment value to obtain the target time step of the diffusion model.

[0072] Exemplarily, the time step offset formula for the target time step is:

[0073]

[0074] Among them, t ′ represents the target time step, α represents the time step offset parameter, and t represents the initial time step.

[0075] In some optional embodiments, during the training and inference process of the diffusion model, the same time step offset parameter and time step offset formula can be used to adjust the time step distribution, thereby reducing the performance degradation caused by the mismatch of the time step distribution and ensuring the stability of the diffusion model generation.

[0076] The technical solution of the embodiment of the present invention determines a first nonlinear time step adjustment value based on a time step offset parameter and an initial time step of a diffusion model; determines a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model; and determines a target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value. The above technical solution generates a target time step adapted to the characteristics of a target video according to the time step offset parameter, and then uses the target time step to perform a video generation task, which can effectively improve the video generation quality and efficiency of the diffusion model.

[0077] Embodiment 4

[0078] Figure 4 This is a schematic diagram of the structure of a time step adjustment device for a diffusion model provided by the fourth embodiment of the present invention. Figure 4 As shown, the device comprises:

[0079] The video resolution and video frame number acquisition module 410 is used to acquire the video resolution and the video frame number of the target video;

[0080] A time step offset parameter determination module 420, configured to determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video;

[0081] The time step nonlinear adjustment module 430 is used to obtain the initial time step of the diffusion model, and based on the time step offset parameter, nonlinearly adjust the initial time step of the diffusion model to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0082] The technical solution of the embodiment of the present invention obtains the video resolution and the number of video frames of the target video; determines the time step offset parameter based on the video resolution and the number of video frames of the target video; obtains the initial time step of the diffusion model, and nonlinearly adjusts the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise from the target video. The above technical solution dynamically calculates the time step offset parameter according to the video resolution and the number of video frames of the target video, and then generates the target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then uses the target time step to perform video generation tasks, which can effectively improve the video generation quality and efficiency of the diffusion model.

[0083] In some optional implementations, the time step offset parameter determination module 420 includes:

[0084] A spatial dimension offset parameter determining unit, used to determine a spatial dimension offset parameter corresponding to the video resolution of the target video;

[0085] A time dimension offset parameter determination unit, used to determine the time dimension offset parameter corresponding to the number of video frames of the target video;

[0086] A time step offset parameter determination unit is used to determine a time step offset parameter based on a spatial dimension offset parameter corresponding to a video resolution of the target video and a time dimension offset parameter corresponding to a number of video frames of the target video.

[0087] In some optional implementations, the spatial dimension offset parameter determination unit may be specifically configured to:

[0088] Based on a preset video resolution-offset mapping relationship, determining a spatial dimension offset parameter corresponding to the video resolution of the target video;

[0089] The preset video resolution-offset mapping relationship includes multiple video resolutions and an offset parameter corresponding to each video resolution.

[0090] In some optional implementations, the time dimension offset parameter determination unit may be specifically configured to:

[0091] Determining the square root of the number of video frames of the target video;

[0092] The square root of the number of video frames of the target video is determined as a time dimension offset parameter corresponding to the number of video frames of the target video.

[0093] In some optional implementations, the time step nonlinear adjustment module 430 includes:

[0094] a first nonlinear time step adjustment value determining unit, configured to determine a first nonlinear time step adjustment value based on the time step offset parameter and an initial time step of the diffusion model;

[0095] a second nonlinear time step adjustment value determining unit, configured to determine a second nonlinear time step adjustment value based on the time step offset parameter and an initial time step of the diffusion model;

[0096] The target time step determining unit is used to determine a target time step of a diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

[0097] In some optional implementations, the first nonlinear time step adjustment value determining unit may be specifically configured to:

[0098] The time step offset parameter is multiplied by an initial time step of the diffusion model to obtain a first nonlinear time step adjustment value.

[0099] In some optional implementations, the second nonlinear time step adjustment value determining unit may be specifically configured to:

[0100] Subtract 1 from the time step offset parameter to obtain a first time step adjustment intermediate value;

[0101] multiplying the first time step adjusted intermediate value and the initial time step of the diffusion model to obtain a second time step adjusted intermediate value;

[0102] The second time step adjustment intermediate value is increased by 1 to obtain a second nonlinear time step adjustment value.

[0103] The time step adjustment device for a diffusion model provided in an embodiment of the present invention can execute the time step adjustment method for a diffusion model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0104] Embodiment 5

[0105] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0106] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0107] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0108] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a time step adjustment method for a diffusion model, which includes:

[0109] Get the video resolution and the number of video frames of the target video;

[0110] Determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video;

[0111] An initial time step of a diffusion model is obtained, and based on the time step offset parameter, a nonlinear adjustment is performed on the initial time step of the diffusion model to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

[0112] In some embodiments, the time step adjustment method of the diffusion model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the time step adjustment method of the diffusion model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the time step adjustment method of the diffusion model in any other appropriate manner (e.g., by means of firmware).

[0113] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0115] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0118] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0119] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A time step adjustment method for a diffusion model, characterized in that: include: Get the video resolution and the number of video frames of the target video; Determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video; An initial time step of a diffusion model is obtained, and based on the time step offset parameter, a nonlinear adjustment is performed on the initial time step of the diffusion model to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

2. The method according to claim 1, characterized in that The determining of the time step offset parameter based on the video resolution of the target video and the number of video frames of the target video comprises: Determine a spatial dimension offset parameter corresponding to the video resolution of the target video; Determine a time dimension offset parameter corresponding to the number of video frames of the target video; A time step offset parameter is determined based on a spatial dimension offset parameter corresponding to the video resolution of the target video and a temporal dimension offset parameter corresponding to the number of video frames of the target video.

3. The method according to claim 2, characterized in that The determining of the spatial dimension offset parameter corresponding to the video resolution of the target video includes: Based on a preset video resolution-offset mapping relationship, determining a spatial dimension offset parameter corresponding to the video resolution of the target video; The preset video resolution-offset mapping relationship includes multiple video resolutions and an offset parameter corresponding to each video resolution.

4. The method according to claim 2, characterized in that: The step of determining the time dimension offset parameter corresponding to the number of video frames of the target video includes: Determining the square root of the number of video frames of the target video; The square root of the number of video frames of the target video is determined as a time dimension offset parameter corresponding to the number of video frames of the target video.

5. The method according to claim 1, characterized in that The nonlinearly adjusting the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model includes: determining a first nonlinear time step adjustment value based on the time step offset parameter and an initial time step of the diffusion model; determining a second nonlinear time step adjustment value based on the time step offset parameter and an initial time step of the diffusion model; A target time step for a diffusion model is determined based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

6. The method according to claim 5, characterized in that The determining of a first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model comprises: The time step offset parameter is multiplied by an initial time step of the diffusion model to obtain a first nonlinear time step adjustment value.

7. The method according to claim 5, characterized in that The determining of a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model comprises: Subtract 1 from the time step offset parameter to obtain a first time step adjustment intermediate value; multiplying the first time step adjusted intermediate value and the initial time step of the diffusion model to obtain a second time step adjusted intermediate value; The second time step adjustment intermediate value is increased by 1 to obtain a second nonlinear time step adjustment value.

8. A time step adjustment device for a diffusion model, characterized in that: include: A video resolution and video frame number acquisition module is used to acquire the video resolution and video frame number of a target video; A time step offset parameter determination module, used to determine a time step offset parameter based on a video resolution of the target video and a number of video frames of the target video; The time step nonlinear adjustment module is used to obtain the initial time step of the diffusion model, and based on the time step offset parameter, nonlinearly adjust the initial time step of the diffusion model to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise into and remove noise from the target video.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the time step adjustment method of the diffusion model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the time step adjustment method of the diffusion model according to any one of claims 1 to 7 when executed.

Images