Content generation method and apparatus, and computing device cluster

Abstract

The content generation method comprises the following steps: in response to acquired first data input by a user, processing an initial signal obtained by random sampling through a diffusion model to obtain a first prediction noise signal; denoising the first prediction noise signal through diffusion model iteration to obtain first content related to the first data; and performing noise addition and removal processing on the first content through diffusion model iteration to obtain second content related to the first data. Thus, through a two-stage content generation mode, the content structure can be quickly constructed by using fewer reasoning times in the first stage, and content details can be prompted in the second stage, so that high-quality content can be generated more efficiently, and the structure details are rich and real.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a content generation method, device and computing device cluster. Background Art

[0002] The diffusion model (DM) is a generative model that performs multi-step iterative denoising on the initial signal obtained by random sampling to find its corresponding meaningful position in the image probability distribution. This position defines the image information, and the denoising process is the image generation process. In addition to image tasks, the diffusion model has also been used for generation tasks in other content forms such as audio, video, and text.

[0003] The training of diffusion models can usually be divided into two processes: forward and reverse. The forward process mainly involves adding noise, while the reverse process mainly involves denoising. In the training phase of the diffusion model, by adding noise to samples such as images, data pairs of noisy samples and noise are obtained. The noisy data is input into the neural network of the reverse process to predict the added noise signal. The distance between the predicted noise and the actual noise is reduced through supervised learning methods to train the noise prediction ability of the diffusion model. Among them, the diffusion model can predict the noise in the sample, so it can denoise the noisy sample. In the inference phase of the diffusion model, the diffusion model is used to iterate multiple times to gradually denoise a randomly sampled initial signal to generate the required data, such as images, audio or video.

[0004] The diffusion model is able to generate higher quality content than generative methods such as variational autoencoder (VAE) and generative adversarial network (GAN) due to its iterative generation method. However, the diffusion model also takes more time and computational overhead to reason due to its iterative generation method. Therefore, how to improve the reasoning speed of the diffusion model is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a content generation method, apparatus, computing device cluster, computer storage medium and computer product, which can improve the reasoning speed of the diffusion model.

[0006] In the first aspect, the present application provides a content generation method, including: in response to the first data input by the user, the initial signal obtained by random sampling is processed by a diffusion model to obtain a first predicted noise signal; the first predicted noise signal is denoised by iterative diffusion model to obtain a first content related to the first data; the first content is subjected to noise addition and removal processing by iterative diffusion model to obtain a second content related to the first data. Exemplarily, the method can be applied to the end side, such as electronic devices such as computers, and can also be applied to the cloud side, such as cloud servers. When the method is applied to the end side, after obtaining the second content, the end side can present the second content to the user. When the method is applied to the cloud side, the first data can be obtained by the end side, and the first data can be transmitted from the end side to the cloud side, so that the cloud side can obtain the first data; after obtaining the second content, the cloud side can transmit the second content to the end side, so as to present the second content to the user on the end side.

[0007] In this way, through the two-stage content generation method, the content structure (such as picture structure, etc.) can be quickly constructed with fewer inference times in the first stage (i.e., the process of obtaining the first content through iteration), and the content details can be prompted in the second stage (i.e., the process of obtaining the second content through iteration), so that high-quality content can be generated more efficiently, making the structural details rich and realistic. Since local details can be improved in the second stage, it is only necessary to construct the content structure in the first stage without constructing local details, so the number of iterations in the first stage can be reduced, and the reduced number of iterations can be less than the number of iterations in the second stage, which reduces the total number of iterations, thereby improving the content generation efficiency while ensuring the quality of the generated content. At the same time, the quality of the generated content can also be improved while the total number of iterations remains unchanged.

[0008] In a possible implementation, the first prediction noise signal is iteratively denoised, including: in the current round of iteration, based on the prediction noise signal generated by k rounds (k≥1) of iterations before the current round, determining the truncation error in the iteration process, the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; using the truncation error, updating the second prediction noise signal obtained in the current round of iteration. In this way, the deviation between the ideal denoising result and the actual denoising result can be estimated through the historical prediction noise signal, so that the deviation can be used to correct the result obtained in the current round of iteration to improve the accuracy of the denoising result.

[0009] In a possible implementation, before using the truncation error to update the second prediction noise signal obtained in the current round of iteration, the method further includes: over-correcting the truncation error. In this way, the over-correction can avoid the smooth transition of the iteratively generated result and retain the local primary texture, so that there is more room for adjustment in the second stage, thereby improving the quality of content generation.

[0010] In one possible implementation, in the process of iteratively adding and removing noise from the first content, the amount of noise added is greater than the amount of noise removed. Since the first stage has made the result converge to the vicinity of the target position (i.e., the optimal denoising result), adding more noise in the second stage can make the result generated in the first stage oscillate near the target position. In the oscillation process, the distance from a noise point (i.e., the noise point) to the target position is generally smaller than the distance between two noise points, so a smaller coefficient is used during denoising to remove a small amount of noise. In this way, the quality of content generation can be improved by adding more noise and removing less noise.

[0011] In a possible implementation, before denoising the first prediction noise signal through the diffusion model iteration, the method further includes: selecting a subsequence of a first length from the inference sequence of the diffusion model, the subsequence being used to characterize the number of times the first prediction noise signal is denoised and the degree of denoising each time, thereby reducing the inference steps of the diffusion model and improving the inference speed.

[0012] In a possible implementation, the number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content. Since the first stage has converged from a random position to near the target position, and in order to facilitate the second stage to quickly fine-tune, the number of iterations in the second stage is relatively small. In this way, the inference speed can be improved and resource overhead can be reduced while ensuring the quality of the generated content.

[0013] In a possible implementation, the method is applied to a server (such as a cloud server, etc.). At this time, before the initial signal obtained by random sampling is processed by the diffusion model, it also includes: obtaining the first data sent by the electronic device; after obtaining the second content related to the first data, it also includes: sending the second content to the electronic device. In this way, when the computing power of the electronic device is insufficient, the required content can be obtained through the cooperation of the end and the cloud.

[0014] In a second aspect, the present application provides a content generation device, comprising: a first processing module and a second processing module. The first processing module is used to process the initial signal obtained by random sampling through a diffusion model in response to the first data input by the user, so as to obtain a first predicted noise signal. The second processing module is used to denoise the first predicted noise signal through diffusion model iteration to obtain first content related to the first data, and to add and remove noise to the first content through diffusion model iteration to obtain second content related to the first data.

[0015] In a possible implementation, when the second processing module iteratively denoises the first prediction noise signal, it is specifically used to: determine the truncation error in the iteration process based on the prediction noise signal generated in k rounds (k≥1) of iterations before the current round, and the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; and use the truncation error to update the second prediction noise signal obtained in the current round of iteration.

[0016] In a possible implementation, before the second processing module uses the truncation error to update the second predicted noise signal obtained in the current round of iteration, it is further configured to: perform overcorrection on the truncation error.

[0017] In a possible implementation, during the process of iteratively performing noise addition and removal processing on the first content, the amount of noise added is greater than the amount of noise removed.

[0018] In a possible implementation, before the second processing module denoises the first prediction noise signal through the diffusion model iteration, it is also used to: filter out a subsequence of a first length from the inference sequence of the diffusion model, the subsequence being used to characterize the number of times the first prediction noise signal is denoised and the degree of denoising each time.

[0019] In a possible implementation manner, the number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

[0020] In a possible implementation, the content generation device may be configured on a server. In this case, the content generation device may further include: a communication module. The communication module is used to obtain the first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling through the diffusion model; and after the second processing module obtains the second content related to the first data, send the second content to the electronic device.

[0021] In a third aspect, the present application provides a content generation device, including a processor and a memory, wherein the memory is used to store computer instructions, and when the computer instructions are executed by the processor, the content generation device executes the method provided in the first aspect. Exemplarily, the content generation device may be, but is not limited to, a chip.

[0022] In a fourth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0023] In a fifth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0024] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation of the first aspect, or, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0025] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the architecture of a content generation system provided in an embodiment of the present application;

[0027] Figure 2 It is a flowchart of a content generation method provided by an embodiment of the present application;

[0028] Figure 3 It is a schematic diagram of a process for correcting a denoising result provided in an embodiment of the present application;

[0029] Figure 4 is a structural diagram of a content generation device provided in an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0032] Figure 7 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0034] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0035] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0036] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0037] For example, Figure 1 FIG. 1 is a schematic diagram showing the architecture of a content generation system provided by an embodiment of the present application. Figure 1 As shown, the content generation system 100 may include: a client 110 and a server 120. The client 110 and the server 120 may communicate with each other via a network, but is not limited to the communication.

[0038] Among them, the client 110 is mainly responsible for receiving the data (such as text, voice, picture, etc.) input by the user, and transmitting the data to the server 220, and receiving the content related to the data input by the user generated from the server 220, such as pictures, videos, audio, etc. Among them, the client 110 can be a desktop application, a mobile application, a Web application or a Web-based application. In this embodiment, the user can enter data such as text on the client 110. After the user completes the input and triggers content generation on the client 110, the client 110 can send the data input by the user to the server 120. Exemplarily, after completing the input, the user can click a button on the client 110 for executing the content generation task, for example, when the client 110 is a Web application, the user can click the "enter" button on the keyboard to trigger the content generation task.

[0039] The server 120 is mainly responsible for receiving data from the client 10, and, based on the received data, generating content related to the data, and sending the content to the client 110. Exemplarily, a diffusion model 121 may be configured in the server 120. The server 120 may generate content related to the data input by the user through the diffusion model 121 configured thereon. In addition, the server 110 is usually deployed on a server and has certain computing and storage resources. Exemplarily, the server on which the server 120 is deployed may be, but is not limited to, a cloud server, a physical server, or a virtual private server.

[0040] The above is an introduction to the content generation system 100 provided in the embodiment of the present application. Next, based on the above content, a content generation method provided in the embodiment of the present application is introduced.

[0041] For example, Figure 2 FIG. 1 is a flow chart of a content generation method provided by an embodiment of the present application. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the content generation method may include the following steps:

[0042] S201 . In response to first data obtained from a user input, an initial signal obtained by random sampling is processed by a diffusion model to obtain a first predicted noise signal.

[0043] In this embodiment, after obtaining the first data input by the user, random sampling can be performed through the diffusion model to obtain an initial signal. Exemplarily, the first data can be text, pictures, or audio, etc. Then, the initial signal obtained by sampling is processed through the diffusion model to obtain a first predicted noise signal.

[0044] S202: De-noise the first prediction noise signal through diffusion model iteration to obtain first content related to the first data.

[0045] In this embodiment, after obtaining the first predicted noise signal, the first predicted noise signal can be denoised by iterative diffusion model to obtain the first content related to the first data, such as: picture, audio, video, etc. Exemplarily, during the iterative process of the diffusion model, the denoising degree of each round of iteration can be different. Exemplarily, the denoising result of the t+1th round can be:

[0046]

[0047] Among them, x t+1 is the denoising result of the t+1th round, x t is the denoising result of the tth round, α and are all hyperparameters, ∈ θ (x t ,t) is the estimated noise of the tth round.

[0048] In some embodiments, in order to improve the accuracy of the denoising results, the results obtained in the current round of iterations may be updated using the results obtained in the previous rounds of iterations. Figure 3 As shown, the following steps may be included:

[0049] S301: Determine a truncation error in an iterative process based on a predicted noise signal generated in k rounds (k≥1) before the current round, wherein the truncation error is used to characterize the deviation between an ideal denoising result and an actual denoising result.

[0050] In this embodiment, the value of k can be preset or randomly generated. In the current round of iteration, the prediction noise signal generated by the k rounds of iteration before the current round (i.e., a part of the historical prediction noise signal or all of the historical prediction noise signal) can be used to determine the truncation error in the iteration process. Exemplarily, the truncation error of the t+1th round can be determined by the following formula:

[0051]

[0052] Among them, r(t,i) is the correction coefficient, x t-i is the result generated by the ti-th iteration, α and are all hyperparameters, ∈ θ (x t ,t) is the estimated noise of the tth round. Exemplarily, r(t,1) to r(t,k) may be a geometric progression.

[0053] S302: Using the truncation error, update the second predicted noise signal obtained in the current round of iteration.

[0054] In this embodiment, after the truncation error is obtained, the truncation error can be used to update the second prediction noise signal obtained in the current round of iteration. From the above "Formula 1" and "Formula 2", it can be obtained that the prediction noise signal generated in the t+1th round is:

[0055]

[0056] After the first content is obtained through iteration, S203 may be executed.

[0057] S203: Perform noise addition and removal processing on the first content through diffusion model iteration to obtain second content related to the first data.

[0058] In this embodiment, after obtaining the first content, the first content can be subjected to noise addition and removal processing through diffusion model iteration to obtain second content related to the first data. In the current round of iteration, noise can be obtained by random sampling, and the noise can be added to the content to be processed, and then denoised again. Thus, by using random resampling to re-degrade the result obtained in the first iteration process and denoising again, local details can be improved to restore more detail information and optimize the quality of the content finally generated.

[0059] In this way, through the two-stage content generation method, the content structure (such as picture structure, etc.) can be quickly constructed in the first stage with fewer inference times, and the content details can be improved in the second stage, so that high-quality content can be generated more efficiently, making the structural details rich and realistic. Since local details can be improved in the second stage, it is only necessary to construct the content structure in the first stage without constructing local details, so the number of iterations in the first stage can be reduced, and the reduced number of iterations can be less than the number of iterations in the second stage, which reduces the total number of iterations, thereby improving the content generation efficiency while ensuring the quality of the generated content. At the same time, the quality of the generated content can also be improved while the total number of iterations remains unchanged.

[0060] In some embodiments, S202 can be understood as a process of restoring low-frequency information, and the first content generated at this time is the overall structure and primary texture of the final content. S203 can be understood as a process of restoring high-frequency information, and at this time, the local texture, contrast, etc. of the content generated by S202 is adjusted to enhance the local details of the content and optimize the content quality.

[0061] For example, when the content required by the user is a face picture, the first content obtained through S202 may be the shape of the face and facial features, and the skin texture of the face. The second content obtained through S203 may be the result of adjusting the local texture and contrast in the first content.

[0062] When the content required by the user is an audio segment, the first content obtained through S202 may be a keynote melody, and the second content obtained through S203 may be some drum beats and lyrics.

[0063] In some embodiments, in the aforementioned S302, the truncation error can be overcorrected, for example, the original correction coefficient is a geometric progression, and different correction coefficients can be multiplied by different values. In this way, the result generated by the first stage iteration can be prevented from being smoothed by overcorrection, and the local primary texture can be retained, so that more adjustment space can be provided in the second stage, so that the second stage can play a greater role, and then the quality of content generation can be further improved. At this time, in the second stage, the coefficient of adding noise can be greater than the coefficient of removing noise, wherein these two coefficients can be hyperparameters. In this way, in the second stage, the amount of noise added is greater than the amount of noise removed. It should be understood that since the first stage has made the result converge to the vicinity of the target position (i.e., the optimal denoising result), adding more noise in the second stage can make the result generated in the first stage oscillate near the target position. In the oscillation process, the distance from a noise point (i.e., the noise point) to the target position is generally less than the distance between two noise points, so a smaller coefficient is used during denoising to remove a small amount of noise.

[0064] In some embodiments, the number of iterations in the second stage (i.e., S203) can be a hyperparameter, which can be 2 or 3. The number of iterations in the first stage (i.e., S202) can also be a hyperparameter, which can be 7 or 8, etc. Exemplarily, since the first stage has converged from a random position to the vicinity of the target position, and in order to facilitate the rapid fine-tuning of the second stage, the number of iterations in the second stage can be less than the number of iterations in the first stage. In this way, the inference speed can be improved and the resource overhead can be reduced while ensuring the quality of the generated content.

[0065] In some embodiments, the diffusion model may have a preset reasoning sequence, which requires different degrees of noise removal in different reasoning steps. For example: when the reasoning sequence is 1 to 1000, when removing noise, it is necessary to start from the 1000th, and the degree of denoising at this time may be the highest, and then decrease in sequence. Exemplarily, the numbers in the reasoning sequence can be understood as the number of steps of reasoning. Further, in order to speed up the reasoning speed of the diffusion model, before the first stage (i.e., S202), a subsequence of the first length can be screened out from the reasoning sequence of the diffusion model. The subsequence can be used to characterize the number of denoising in the first stage (i.e., the number of iterations) and the degree of denoising each time. Exemplarily, a subsequence can be uniformly sampled from the reasoning sequence. For example, when the reasoning sequence is 1 to 1000, if the first length is 10, the subsequence can be 1000, 900, 800, 700, 600, 500, 400, 300, 200, 100.

[0066] It is understandable that the above Figure 2 The content generation method shown in the figure can be executed by the terminal side (such as a computer or other electronic device) or by the cloud side (such as a server, cloud server, etc.). When executed by the terminal side, after the terminal side obtains the second content, the second content can be presented to the user. When executed by the cloud side, the terminal side can obtain the first data input by the user, and then transmit the first data to the cloud side, so that the cloud side can obtain the first data from the terminal side and process the first data; after obtaining the second content, the cloud side can transmit the second content to the terminal side, so that the second content can be presented to the user on the terminal side.

[0067] It is understandable that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0068] The above is an introduction to the content generation method provided in the embodiment of the present application. Next, based on the method in the above embodiment, a content generation device provided in the embodiment of the present application is introduced.

[0069] For example, Figure 4 FIG. 1 is a schematic diagram showing the structure of a content generation device provided by an embodiment of the present application. Figure 4As shown, a first processing module 401 and a second processing module 402. The first processing module 401 is used to process the initial signal obtained by random sampling through a diffusion model in response to the first data input by the user, so as to obtain a first predicted noise signal. The second processing module 402 is used to denoise the first predicted noise signal through diffusion model iteration to obtain first content related to the first data; and to add and remove noise to the first content through diffusion model iteration to obtain second content related to the first data.

[0070] In some embodiments, when iteratively denoising the first predicted noise signal, the second processing module 402 is specifically used to: during the current round of iterations, based on the predicted noise signal generated in k rounds (k≥1) of iterations before the current round, determine the truncation error in the iteration process, the truncation error being used to characterize the deviation between the ideal denoising result and the actual denoising result; and use the truncation error to update the second predicted noise signal obtained in the current round of iterations.

[0071] In some embodiments, before using the truncation error to update the second predicted noise signal obtained in the current round of iteration, the second processing module 402 is further configured to: perform overcorrection on the truncation error.

[0072] In some embodiments, during the process of iteratively adding and removing noise from the first content, the amount of noise added is greater than the amount of noise removed.

[0073] In some embodiments, before denoising the first prediction noise signal through the diffusion model iteration, the second processing module 402 is further used to: filter out a subsequence of a first length from the inference sequence of the diffusion model, the subsequence being used to characterize the number of times the first prediction noise signal is denoised and the degree of denoising each time.

[0074] In some embodiments, the number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

[0075] In some embodiments, the content generation device 400 is configured on a server. In this case, the content generation device 400 may further include: a communication module (not shown in the figure). The communication module is used to obtain the first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling through the diffusion model; and after the second processing module obtains the second content related to the first data, send the second content to the electronic device.

[0076] In some embodiments, Figure 4The first processing module 401 and the second processing module 402 shown in the figure can be implemented by software or by hardware. Exemplarily, the implementation of the first processing module 401 is described below by taking the first processing module 401 as an example. Similarly, the implementation of the second processing module 402 can refer to the implementation of the first processing module 401, and the implementation of the communication module not shown can also refer to the implementation of the first processing module 401.

[0077] As an example of a software functional unit, the first processing module 401 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first processing module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region can include multiple AZs.

[0078] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0079] As an example of a hardware functional unit, the first processing module 401 may include at least one computing device, such as a server, etc. Alternatively, the first processing module 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0080] The multiple computing devices included in the first processing module 401 can be distributed in the same region or in different regions. The multiple computing devices included in the first processing module 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first processing module 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0081] It should be noted that, in other embodiments, the first processing module 401 can be used to execute any step in the content generation method described in the above embodiment, and the second processing module 402 can be used to execute any step in the content generation method described in the above embodiment. The steps that the first processing module 401 and the second processing module 402 are responsible for implementing can be specified as needed, and the first processing module 401 and the second processing module 402 respectively implement different steps in the content generation method described in the above embodiment to achieve Figure 4 The overall functionality of the content generation device 400 is shown.

[0082] The present application also provides a computing device 500. Figure 5 As shown, computing device 500 includes: bus 502, processor 504, memory 506 and communication interface 508. Processor 504, memory 506 and communication interface 508 communicate through bus 502. Computing device 500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 500.

[0083] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 504 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 504 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0084] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0085] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0086] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the aforementioned Figure 4 The functions of the first processing module 401 and the second processing module 402 shown in the embodiment are implemented to realize the content generation method described in the above embodiment. That is, the memory 506 stores instructions for executing the content generation method described in the above embodiment.

[0087] Alternatively, the memory 506 stores executable codes, and the processor 504 executes the executable codes to respectively implement the aforementioned Figure 4 The functions of the content generating apparatus 400 shown in the embodiment are implemented to realize the content generating method described in the above embodiment. That is, the memory 506 stores instructions for executing the content generating method described in the above embodiment.

[0088] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0089] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0090] like Figure 6 As shown, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the content generation method described in the above embodiment.

[0091] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the content generation method described in the above embodiment. In other words, the combination of one or more computing devices 500 can jointly execute instructions for executing the content generation method described in the above embodiment.

[0092] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 4 The content generation apparatus 400 shown in FIG. 5 is a partial function. That is, the instructions stored in the memory 506 in different computing devices 500 can implement the functions of one or more modules in the first processing module 401 and the second processing module 402 .

[0093] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 7 A possible implementation is shown. Figure 7 As shown, two computing devices 500A and 500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the first processing module 501. At the same time, the memory 506 in the computing device 500B stores instructions for executing the functions of the second processing module 502.

[0094] It should be understood that Figure 7The functions of the computing device 500A shown in FIG. 5A may also be completed by multiple computing devices 500. Similarly, the functions of the computing device 500B may also be completed by multiple computing devices 500.

[0095] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 6 and Figure 7 The connection mode of the computing device cluster is different in that the memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the method in the above embodiment.

[0096] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 500 may jointly execute instructions for executing the aforementioned data processing method.

[0097] Based on the method in the above embodiment, the embodiment of the present application provides a content generation device, including a processor and a memory, the memory is used to store computer instructions, when the computer instructions are executed by the processor, Figure 4 The content generation device 400 shown in the figure performs the method in the above embodiment. Exemplarily, the content generation device may be, but is not limited to, a chip.

[0098] Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0099] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment, or, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above embodiment.

[0100] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0101] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0103] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A content generation method, characterized in that: include: In response to the acquired first data input by the user, an initial signal obtained by random sampling is processed by a diffusion model to obtain a first predicted noise signal; Iteratively denoising the first predicted noise signal through the diffusion model to obtain first content related to the first data; The first content is iteratively processed by adding and removing noise through the diffusion model to obtain second content related to the first data.

2. The method according to claim 1, characterized in that Iteratively denoising the first prediction noise signal, comprising: In the current round of iteration, based on the predicted noise signal generated in k rounds (k≥1) of iteration before the current round, a truncation error in the iteration is determined, wherein the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; The second predicted noise signal obtained in the current round of iteration is updated using the truncation error.

3. The method according to claim 2, characterized in that Before using the truncation error to update the second predicted noise signal obtained in the current round of iteration, the method further includes: The truncation error is overcorrected.

4. The method according to claim 3, characterized in that In the process of iteratively performing noise adding and removing processing on the first content, the amount of noise added is greater than the amount of noise removed.

5. The method according to any one of claims 1 to 4, characterized in that: Before denoising the first predicted noise signal through the diffusion model iteration, the method further includes: A subsequence of a first length is selected from the inference sequence of the diffusion model, and the subsequence is used to represent the number of times the first prediction noise signal is denoised and the degree of denoising each time.

6. The method according to any one of claims 1 to 5, characterized in that: The number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to a server; Before the initial signal obtained by random sampling is processed by the diffusion model, it also includes: Acquire the first data sent by the electronic device; After obtaining the second content related to the first data, the method further includes: The second content is sent to the electronic device.

8. A content generating device, characterized in that: include: A first processing module, configured to process an initial signal obtained by random sampling through a diffusion model in response to first data input by a user, to obtain a first predicted noise signal; a second processing module, configured to perform denoising on the first prediction noise signal through iteration of the diffusion model to obtain first content related to the first data; The second processing module is further configured to perform noise addition and removal processing on the first content through iteration of the diffusion model to obtain second content related to the first data.

9. The device according to claim 8, characterized in that When the second processing module iteratively denoises the first predicted noise signal, it is specifically used to: In the current round of iteration, based on the predicted noise signal generated in k rounds (k≥1) of iteration before the current round, a truncation error in the iteration is determined, wherein the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; The second predicted noise signal obtained in the current round of iteration is updated using the truncation error.

10. The device according to claim 9, characterized in that Before the second processing module updates the second predicted noise signal obtained by the current round of iteration using the truncation error, it is further configured to: The truncation error is overcorrected.

11. The device according to claim 10, characterized in that In the process of iteratively performing noise adding and removing processing on the first content, the amount of noise added is greater than the amount of noise removed.

12. The device according to any one of claims 8 to 11, characterized in that: Before denoising the first predicted noise signal through the diffusion model iteration, the second processing module is further used to: A subsequence of a first length is selected from the inference sequence of the diffusion model, and the subsequence is used to represent the number of times the first prediction noise signal is denoised and the degree of denoising each time.

13. The device according to any one of claims 8 to 12, characterized in that: The number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

14. The device according to any one of claims 8 to 13, characterized in that: The device is configured on a server, and the device further comprises: a communication module; The communication module is used to obtain the first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling through the diffusion model; and after the second processing module obtains the second content related to the first data, send the second content to the electronic device.

15. A content generating device, characterized in that: The device comprises a processor and a memory, wherein the memory is used to store computer instructions. When the computer instructions are executed by the processor, the content generating device executes the method according to any one of claims 1 to 7.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the instructions are executed by a computing device cluster, the computing device cluster causes the computing device cluster to perform the method according to any one of claims 1 to 7, wherein the computing device cluster comprises at least one computing device.

18. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster includes at least one computing device.

Images