Time step adjustment method and device of diffusion model, electronic equipment and storage medium

Abstract

The application discloses a diffusion model time step adjustment method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining the video resolution of a target video and the video frame number of the target video; determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video; obtaining an initial time step of a diffusion model, and performing nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise from the target video. According to the above technical solution, the time step offset parameter is dynamically calculated according to the video resolution and the video frame number of the target video, and then the target time step suitable for the characteristics of the target video is generated according to the time step offset parameter, and when the video generation task is executed by using the target time step, the video generation quality and efficiency of the diffusion model can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a time step adjustment method and device of a diffusion model, an electronic device and a storage medium. BACKGROUND

[0002] In a video generation task, the time step of a diffusion model can directly affect the video generation quality and noise processing efficiency. Specifically, the time step is used to guide the noise injection and denoising process, which defines the noise intensity and signal proportion of the diffusion model in each stage, and is the core parameter of the video generation process.

[0003] In the prior art, the design of the time step is usually based on the following two methods:

[0004] A fixed linear time step sequence is taken, which performs noise injection and denoising according to a linearly uniform time step. This method is simple and easy to use, but cannot be adjusted adaptively according to the characteristics of the input video, which may result in low generation efficiency or unstable quality.

[0005] A predefined distribution sampling is taken, which generates a time step using a Gaussian distribution or other fixed distribution, such as sampling a time step through random noise in the model training stage to enhance the generalization ability of the model. However, this method may result in a mismatch between the time step distribution and the video characteristics under diversified input conditions.

[0006] In the process of implementing the present application, it is found that the prior art has at least the following technical problems: the prior art has the problems of low video generation quality and efficiency. SUMMARY

[0007] The present application provides a time step adjustment method and device of a diffusion model, an electronic device and a storage medium to improve the video generation quality and efficiency.

[0008] According to an aspect of the present application, a time step adjustment method of a diffusion model is provided, comprising:

[0009] obtaining the video resolution of a target video and the video frame number of the target video;

[0010] determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video;

[0011] obtaining an initial time step of a diffusion model, and performing nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise for the target video.

[0012] According to another aspect of the present application, there is provided a device for adjusting a time step of a diffusion model, comprising:

[0013] a video resolution and video frame number obtaining module configured to obtain a video resolution of a target video and a video frame number of the target video;

[0014] a time step offset parameter determining module configured to determine a time step offset parameter based on the video resolution of the target video and the video frame number of the target video;

[0015] a time step nonlinear adjusting module configured to obtain an initial time step of the diffusion model, and perform nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is configured to control the diffusion model to inject and remove noise from the target video.

[0016] According to another aspect of the present application, there is provided an electronic device, comprising:

[0017] at least one processor;

[0018] and a memory connected in communication with the at least one processor;

[0019] wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for adjusting a time step of a diffusion model according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to implement the method for adjusting a time step of a diffusion model according to any one of the embodiments of the present application when executed by the processor.

[0021] The technical solution of the embodiments of the present application comprises the following steps: obtaining a video resolution of a target video and a video frame number of the target video; determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video; obtaining an initial time step of a diffusion model, and performing nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is configured to control the diffusion model to inject and remove noise from the target video. The above technical solution dynamically calculates a time step offset parameter according to the video resolution and the video frame number of the target video, and then generates a target time step suitable for the characteristics of the target video according to the time step offset parameter, and when the target time step is used to perform a video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

[0022] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. Rather, the scope of the embodiments of the application is to be defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0024] Figure 1 is a flow chart of a time step adjustment method of a diffusion model according to an embodiment of the present application;

[0025] Figure 2 is a flow chart of a time step adjustment method of a diffusion model according to an embodiment of the present application;

[0026] Figure 3 is a flow chart of a time step adjustment method of a diffusion model according to an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of a time step adjustment device of a diffusion model according to an embodiment of the present application;

[0028] Figure 5 is a structural schematic diagram of an electronic device for implementing the time step adjustment method of a diffusion model according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should belong to the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a time-step adjustment method for a diffusion model provided in Embodiment 1 of the present invention. This embodiment is applicable to noise sampling training and inference of a diffusion model in a video generation task. The method can be executed by a time-step adjustment device for the diffusion model, which can be implemented in hardware and / or software. This time-step adjustment device can be configured in electronic devices such as terminals and / or servers. Figure 1 As shown, the method includes:

[0033] S110. Obtain the video resolution and the number of video frames of the target video.

[0034] In this embodiment of the invention, the target video refers to the video to be input into the diffusion model.

[0035] For example, in response to parameter input operations, the video resolution and the number of video frames of the target video are obtained.

[0036] S120. Determine the time step offset parameter based on the video resolution and the number of video frames of the target video.

[0037] S130. Obtain the initial time step of the diffusion model, and perform nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject noise and remove noise from the target video.

[0038] In the embodiment of the application, the time step offset parameter refers to an offset parameter for nonlinearly adjusting an initial time step, wherein the initial time step can be an initialized uniform distribution time step or a randomly generated time step, which is not limited here.

[0039] It can be understood that, in the training process of the diffusion model, by injecting and removing noise, the diffusion model can learn how to extract and reconstruct a real video from a noisy video.

[0040] It should be noted that, by dynamically calculating the time step offset parameter according to the video resolution and the video frame number of the target video, and then generating the target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then using the target time step to perform the video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

[0041] The technical scheme of the embodiment of the application comprises the following steps: obtaining a video resolution of a target video and a video frame number of the target video; determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video; obtaining an initial time step of a diffusion model, and nonlinearly adjusting the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject and remove noise from the target video. The above technical scheme can effectively improve the video generation quality and efficiency of the diffusion model by dynamically calculating the time step offset parameter according to the video resolution and the video frame number of the target video, and then generating the target time step that adapts to the characteristics of the target video according to the time step offset parameter, and then using the target time step to perform the video generation task.

[0042] Embodiment two

[0043] Figure 2 A flowchart of a diffusion model time step adjustment method provided by the second embodiment of the application, the method of the present embodiment can be combined with each optional scheme in the diffusion model time step adjustment method provided in the above embodiments. The diffusion model time step adjustment method provided by the present embodiment is further optimized. Optionally, determining the time step offset parameter based on the video resolution of the target video and the video frame number of the target video comprises: determining a spatial dimension offset parameter corresponding to the video resolution of the target video; determining a time dimension offset parameter corresponding to the video frame number of the target video; and determining the time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the time dimension offset parameter corresponding to the video frame number of the target video.

[0044] As Figure 2 shown, the method comprises:

[0045] S210, obtaining a video resolution of a target video and a video frame number of the target video.

[0046] S220, determine a spatial dimension offset parameter corresponding to the video resolution of the target video.

[0047] In the embodiment of the present application, the spatial dimension offset parameter refers to an offset parameter calculated by the video resolution.

[0048] Optionally, determining the spatial dimension offset parameter corresponding to the video resolution of the target video comprises: determining the spatial dimension offset parameter corresponding to the video resolution of the target video based on a preset video resolution-offset mapping relationship; wherein the preset video resolution-offset mapping relationship comprises a plurality of video resolutions and an offset parameter corresponding to each video resolution.

[0049] Illustratively, the range of the offset parameter is (A, B), A represents a basic offset parameter, and B represents a maximum offset parameter. Specifically, the video resolution can be mapped to (A, B), and the spatial dimension offset parameter is matched from (A, B). It should be noted that the higher the video resolution, the closer the offset parameter to the maximum offset parameter, so that the time step is adjusted by the offset parameter corresponding to the high video resolution, so that the time step distribution is offset to the high signal area, wherein the high signal area refers to a high resolution long video. The lower the video resolution, the closer the offset parameter to the basic offset parameter, so that the time step is adjusted by the offset parameter corresponding to the low video resolution, so that the time step distribution is more uniform.

[0050] It should be noted that the embodiment of the present application can concentrate the time step in the high signal area by dynamically adjusting the time step, enhance the detail preservation capability, and adapt to the high resolution long video generation requirement. In addition, it can also reduce the calculation of the low signal area, improve the video generation efficiency, avoid redundant calculation waste, and adapt to the low resolution short video generation requirement.

[0051] S230, determine a time dimension offset parameter corresponding to the video frame number of the target video.

[0052] In the embodiment of the present application, the time dimension offset parameter refers to an offset parameter calculated by the video frame number.

[0053] Specifically, the square root of the video frame number of the target video can be determined; and the square root of the video frame number of the target video is determined as the time dimension offset parameter corresponding to the video frame number of the target video.

[0054] S240, determining a time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the time dimension offset parameter corresponding to the video frame number of the target video.

[0055] Exemplarily, the spatial dimension offset parameter corresponding to the video resolution of the target video is multiplied by the time dimension offset parameter corresponding to the video frame number of the target video to obtain the time step offset parameter.

[0056] S250, obtain an initial time step of the diffusion model, perform nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject and remove noise for the target video.

[0057] The technical scheme of the embodiment of the application determines the spatial dimension offset parameter corresponding to the video resolution of the target video, then determines the time dimension offset parameter corresponding to the video frame number of the target video, and then determines the time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the time dimension offset parameter corresponding to the video frame number of the target video, so that dynamic calculation of the time step offset parameter is realized, thereby adapting to diversified video generation tasks.

[0058] Embodiment three

[0059] Figure 3 A flowchart of a diffusion model time step adjustment method provided by the third embodiment of the application, the method of the present embodiment can be combined with each optional scheme in the diffusion model time step adjustment method provided in the above embodiments. The diffusion model time step adjustment method provided by the present embodiment is further optimized. Optionally, the nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model comprises: determining a first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model; determining a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model; and determining the target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

[0060] As Figure 3 shown, the method comprises:

[0061] S310, obtain the video resolution of the target video and the video frame number of the target video.

[0062] S320, determine the time step offset parameter based on the video resolution of the target video and the video frame number of the target video.

[0063] S330, obtain an initial time step of the diffusion model.

[0064] S340, determine a first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model.

[0065] In the embodiments of the present application, the first nonlinear time step adjustment value is a time step adjustment value calculated by the time step offset parameter and the initial time step of the diffusion model.

[0066] Specifically, the time step offset parameter is multiplied by the initial time step of the diffusion model to obtain the first nonlinear time step adjustment value.

[0067] S350, determining a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model.

[0068] In the embodiments of the present application, the second nonlinear time step adjustment value refers to another time step adjustment value calculated by the time step offset parameter and the initial time step of the diffusion model.

[0069] Specifically, the time step offset parameter is subtracted by 1 to obtain a first time step adjustment intermediate value; the first time step adjustment intermediate value is multiplied by the initial time step of the diffusion model to obtain a second time step adjustment intermediate value; and the second time step adjustment intermediate value is added by 1 to obtain the second nonlinear time step adjustment value.

[0070] S360, determining a target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value, wherein the target time step is used to control the diffusion model to inject and remove noise to the target video.

[0071] Specifically, the first nonlinear time step adjustment value is divided by the second nonlinear time step adjustment value to obtain the target time step of the diffusion model.

[0072] Exemplarily, the time step offset formula of the target time step is:

[0073]

[0074] wherein t ′ represents the target time step, a represents the time step offset parameter, and t represents the initial time step.

[0075] In some optional embodiments, during the training and inference process of the diffusion model, the same time step offset parameter and time step offset formula can be used to adjust the time step distribution, so that the performance decline caused by the mismatch of the time step distribution can be reduced, and the stability of the diffusion model can be ensured.

[0076] The technical scheme of the embodiment of the present application determines a first nonlinear time step adjustment value based on the initial time step of the diffusion model and the time step offset parameter; determines a second nonlinear time step adjustment value based on the initial time step of the diffusion model and the time step offset parameter; and determines the target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value. According to the above technical scheme, the target time step adaptive to the characteristics of the target video is generated based on the time step offset parameter, and when the target time step is used to perform the video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

[0077] Embodiment four

[0078] Figure 4 A structural schematic diagram of a diffusion model time step adjustment device provided by the fourth embodiment of the present application is shown in FIG. 4. As shown in the figure, the device comprises: Figure 4 A video resolution and video frame number acquisition module 410 is configured to acquire the video resolution of a target video and the video frame number of the target video.

[0079] A time step offset parameter determination module 420 is configured to determine a time step offset parameter based on the video resolution of the target video and the video frame number of the target video.

[0080] A time step nonlinear adjustment module 430 is configured to acquire an initial time step of a diffusion model, and perform nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject and remove noise from the target video.

[0081] The technical scheme of the embodiment of the present application acquires the video resolution of a target video and the video frame number of the target video; determines a time step offset parameter based on the video resolution of the target video and the video frame number of the target video; acquires an initial time step of a diffusion model, and performs nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject and remove noise from the target video. According to the above technical scheme, the time step offset parameter is dynamically calculated based on the video resolution and the video frame number of the target video, and then the target time step adaptive to the characteristics of the target video is generated based on the time step offset parameter, and when the target time step is used to perform the video generation task, the video generation quality and efficiency of the diffusion model can be effectively improved.

[0082] In some optional embodiments, the time step offset parameter determination module 420 comprises:

[0083]

[0084] ​a spatial dimension offset parameter determination unit, configured to determine a spatial dimension offset parameter corresponding to a video resolution of the target video;

[0085] a time dimension offset parameter determination unit, configured to determine a time dimension offset parameter corresponding to a video frame number of the target video;

[0086] a time step offset parameter determination unit, configured to determine a time step offset parameter based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the time dimension offset parameter corresponding to the video frame number of the target video.

[0087] In some optional embodiments, the spatial dimension offset parameter determination unit can be specifically configured to:

[0088] determine the spatial dimension offset parameter corresponding to the video resolution of the target video based on a preset video resolution-offset mapping relationship;

[0089] In the preset video resolution-offset mapping relationship, a plurality of video resolutions and offset parameters corresponding to each video resolution are included.

[0090] In some optional embodiments, the time dimension offset parameter determination unit can be specifically configured to:

[0091] determine a square root of the video frame number of the target video;

[0092] determine the square root of the video frame number of the target video as the time dimension offset parameter corresponding to the video frame number of the target video.

[0093] In some optional embodiments, the time step nonlinear adjustment module 430 includes:

[0094] a first nonlinear time step adjustment value determination unit, configured to determine a first nonlinear time step adjustment value based on the time step offset parameter and an initial time step of the diffusion model;

[0095] a second nonlinear time step adjustment value determination unit, configured to determine a second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model;

[0096] a target time step determination unit, configured to determine a target time step of the diffusion model based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

[0097] In some optional embodiments, the first nonlinear time step adjustment value determination unit can be specifically configured to:

[0098] multiply the time step offset parameter and the initial time step of the diffusion model to obtain the first nonlinear time step adjustment value.

[0099] In some optional embodiments, the second non-linear time step adjustment value determination unit can be specifically used for:

[0100] subtracting 1 from the time step offset parameter to obtain a first time step adjustment intermediate value;

[0101] multiplying the first time step adjustment intermediate value and the initial time step of the diffusion model to obtain a second time step adjustment intermediate value;

[0102] adding 1 to the second time step adjustment intermediate value to obtain a second non-linear time step adjustment value.

[0103] The diffusion model time step adjustment device provided in the embodiments of the present application can execute the diffusion model time step adjustment method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0104] Embodiment five

[0105] Figure 5 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0106] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An I / O interface 15 is also connected to the bus 14.

[0107] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0108] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the time step adjustment method of the diffusion model, which includes:

[0109] obtaining a video resolution of a target video and a video frame number of the target video;

[0110] determining a time step offset parameter based on the video resolution of the target video and the video frame number of the target video;

[0111] obtaining an initial time step of a diffusion model, performing a non-linear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain a target time step of the diffusion model, wherein the target time step is used to control the diffusion model to inject and remove noise to the target video.

[0112] In some embodiments, the time step adjustment method of the diffusion model can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the time step adjustment method of the diffusion model described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the time step adjustment method of the diffusion model by any other appropriate means, such as by means of firmware.

[0113] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0114] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0115] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0117] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0118] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0119] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0120] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A time step adjustment method for a diffusion model, characterized in that, include: Obtain the video resolution and the number of frames in the target video; The time step offset parameter is determined based on the video resolution and the number of video frames of the target video. The initial time step of the diffusion model is obtained. Based on the time step offset parameter, the initial time step of the diffusion model is nonlinearly adjusted to obtain the target time step of the diffusion model. The target time step is used to control the diffusion model to inject noise and remove noise from the target video.

2. The method according to claim 1, characterized in that, The determination of the time step offset parameters based on the video resolution and the number of video frames of the target video includes: Determine the spatial dimension offset parameter corresponding to the video resolution of the target video; Determine the time dimension offset parameter corresponding to the number of video frames of the target video; The time step offset parameter is determined based on the spatial dimension offset parameter corresponding to the video resolution of the target video and the temporal dimension offset parameter corresponding to the number of video frames of the target video.

3. The method according to claim 2, characterized in that, The step of determining the spatial dimension offset parameter corresponding to the video resolution of the target video includes: Based on the preset video resolution-offset mapping relationship, the spatial dimension offset parameter corresponding to the video resolution of the target video is determined; The preset video resolution-offset mapping relationship includes multiple video resolutions and the offset parameter corresponding to each video resolution.

4. The method according to claim 2, characterized in that, The step of determining the time dimension offset parameter corresponding to the video frame number of the target video includes: Determine the square root of the number of video frames in the target video; The square root of the number of video frames in the target video is determined as the time dimension offset parameter corresponding to the number of video frames in the target video.

5. The method according to claim 1, characterized in that, The step of nonlinearly adjusting the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model includes: The first nonlinear time step adjustment value is determined based on the time step offset parameter and the initial time step of the diffusion model; The second nonlinear time step adjustment value is determined based on the time step offset parameter and the initial time step of the diffusion model; The target time step of the diffusion model is determined based on the first nonlinear time step adjustment value and the second nonlinear time step adjustment value.

6. The method according to claim 5, characterized in that, The step of determining the first nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model includes: The first nonlinear time step adjustment value is obtained by multiplying the time step offset parameter and the initial time step of the diffusion model.

7. The method according to claim 5, characterized in that, The determination of the second nonlinear time step adjustment value based on the time step offset parameter and the initial time step of the diffusion model includes: Subtract 1 from the time step offset parameter to obtain the first time step adjustment intermediate value; The first time step adjustment intermediate value is multiplied by the initial time step of the diffusion model to obtain the second time step adjustment intermediate value; Add 1 to the intermediate value of the second time step adjustment to obtain the second nonlinear time step adjustment value.

8. A time-step adjustment device for a diffusion model, characterized in that, include: The video resolution and video frame rate acquisition module is used to acquire the video resolution and video frame rate of the target video. The time step offset parameter determination module is used to determine the time step offset parameter based on the video resolution of the target video and the number of video frames of the target video. The time-step nonlinear adjustment module is used to obtain the initial time step of the diffusion model, and to perform nonlinear adjustment on the initial time step of the diffusion model based on the time step offset parameter to obtain the target time step of the diffusion model. The target time step is used to control the diffusion model to inject noise and remove noise from the target video.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the time-step adjustment method for the diffusion model according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the time-step adjustment method of the diffusion model according to any one of claims 1-7.

Images