Content generation method and apparatus, and computing device cluster

Abstract

A content generation method, comprising: in response to acquired first data that is input by a user, by means of a diffusion model, processing an initial signal obtained by means of random sampling to obtain a first predicted noise signal (S201); by means of the diffusion model, iteratively denoising the first predicted noise signal to obtain first content related to the first data (S202); and by means of the diffusion model, iteratively performing noise addition and removal processing on the first content to obtain second content related to the first data (S203). Thus, by means of a two-stage content generation method, a content structure can be quickly constructed by using fewer instances of inference in a first stage, and content details can be prompted in a second stage, such that high-quality content can be generated more efficiently, and thus structural details are rich and realistic.

Description

Content generation method, device and computing device cluster

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 14, 2023, with application number 202311192131.X and application name “A content generation method, device and computing device cluster”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a content generation method, apparatus, and computing device cluster. Background Art

[0003] The diffusion model (DM) is a generative model that iteratively denoises a randomly sampled initial signal in multiple steps to find its corresponding meaningful location in the image probability distribution. This location defines the image information, and the denoising process is the image generation process. In addition to image tasks, the diffusion model has also been applied to other content generation tasks such as audio, video, and text.

[0004] Diffusion model training typically consists of two steps: forward and reverse. The forward step primarily involves adding noise, while the reverse step primarily involves denoising. During the training phase, noise is added to samples such as images to generate data pairs of noisy samples and noise. This noisy data is then fed into a neural network in the reverse phase to predict the added noise signal. Supervised learning is used to narrow the gap between the predicted noise and the actual noise, thereby training the diffusion model's noise prediction capabilities. The diffusion model's ability to predict noise in samples allows it to denoise noisy samples. During the inference phase, the diffusion model is used to iteratively denoise a randomly sampled initial signal, generating the desired data, such as images, audio, or video.

[0005] The diffusion model's iterative generation method enables it to generate higher-quality content than generative methods such as variational autoencoders (VAEs) and generative adversarial networks (GANs). However, this iterative generation method also increases the time and computational overhead of the diffusion model's inference process. Therefore, improving the diffusion model's inference speed is a pressing technical challenge.

[0006] Summary of the Invention

[0007] The present application provides a content generation method, apparatus, computing device cluster, computer storage medium, and computer product that can improve the inference speed of a diffusion model.

[0008] In a first aspect, the present application provides a content generation method, comprising: in response to first data input by a user, processing an initial signal obtained by random sampling through a diffusion model to obtain a first predicted noise signal; iteratively denoising the first predicted noise signal through the diffusion model to obtain first content related to the first data; and iteratively adding and removing noise from the first content through the diffusion model to obtain second content related to the first data. Exemplarily, the method can be applied to a terminal side, such as an electronic device such as a computer, or to a cloud side, such as a cloud server. When the method is applied to the terminal side, after obtaining the second content, the terminal side can present the second content to the user. When the method is applied to the cloud side, the first data can be obtained by the terminal side, and then the first data can be transmitted from the terminal side to the cloud side, so that the cloud side can obtain the first data; after obtaining the second content, the cloud side can transmit the second content to the terminal side, so that the second content can be presented to the user on the terminal side.

[0009] In this way, through a two-stage content generation approach, the content structure (e.g., image structure, etc.) can be quickly constructed using fewer inferences in the first stage (i.e., the process of obtaining the first content through iteration), and the content details can be prompted in the second stage (i.e., the process of obtaining the second content through iteration), thereby enabling more efficient generation of high-quality content with rich and realistic structural details. Since local details can be improved in the second stage, only the content structure needs to be constructed in the first stage without constructing local details, so the number of iterations in the first stage can be reduced, and the reduced number of iterations can be less than the number of iterations in the second stage, which reduces the total number of iterations, thereby improving content generation efficiency while ensuring the quality of the generated content. At the same time, the quality of the generated content can also be improved while the total number of iterations remains unchanged.

[0010] In one possible implementation, iteratively denoising a first predicted noise signal includes: determining, during a current iteration, a truncation error based on the predicted noise signal generated in k iterations (k ≥ 1) prior to the current iteration, where the truncation error represents the deviation between an ideal denoising result and an actual denoising result; and using the truncation error to update a second predicted noise signal obtained in the current iteration. In this way, the deviation between the ideal denoising result and the actual denoising result can be estimated using the historical predicted noise signal, and this deviation can be used to correct the result obtained in the current iteration to improve the accuracy of the denoising result.

[0011] In one possible implementation, before using the truncation error to update the second predicted noise signal obtained in the current iteration, the process further includes overcorrecting the truncation error. This overcorrection can prevent the iteratively generated result from becoming excessively smooth and preserve local primary textures, thereby providing more room for adjustment in the second stage and improving the quality of generated content.

[0012] In one possible implementation, in the process of iteratively adding and removing noise from the first content, the amount of noise added is greater than the amount of noise removed. Since the first stage has made the result converge to the vicinity of the target position (i.e., the optimal denoising result), adding more noise in the second stage can make the result generated in the first stage oscillate near the target position. In the oscillation process, the distance from a noise point (i.e., a noise point) to the target position is generally smaller than the distance between two noise points, so a smaller coefficient is used in denoising to remove a small amount of noise. In this way, the quality of content generation can be improved by adding more noise and removing less noise.

[0013] In one possible implementation, before iteratively denoising the first predicted noise signal using the diffusion model, the method further includes: selecting a subsequence of a first length from the inference sequence of the diffusion model, wherein the subsequence is used to represent the number of times the first predicted noise signal is denoised and the degree of denoising each time. This reduces the number of inference steps of the diffusion model and improves inference speed.

[0014] In one possible implementation, the number of iterations required to generate the first content is greater than the number of iterations required to generate the second content. Because the first phase has already converged from a random position to near the target position, and to facilitate rapid fine-tuning in the second phase, the second phase has fewer iterations. This improves inference speed and reduces resource overhead while maintaining the quality of the generated content.

[0015] In one possible implementation, the method is applied to a server (such as a cloud server). Before processing the initial signal obtained by random sampling using a diffusion model, the method further includes: obtaining first data sent by the electronic device; and after obtaining second content related to the first data, the method further includes: sending the second content to the electronic device. In this way, if the computing power of the electronic device is insufficient, the required content can be obtained through end-cloud collaboration.

[0016] In a second aspect, the present application provides a content generation device comprising: a first processing module and a second processing module. The first processing module is configured to, in response to first data input by a user, process an initial signal obtained by random sampling using a diffusion model to obtain a first predicted noise signal. The second processing module is configured to iteratively denoise the first predicted noise signal using the diffusion model to obtain first content related to the first data, and iteratively add and remove noise from the first content using the diffusion model to obtain second content related to the first data.

[0017] In one possible implementation, when iteratively denoising the first predicted noise signal, the second processing module is specifically used to: determine, during a current round of iteration, a truncation error in the iterative process based on the predicted noise signal generated in k rounds (k≥1) of iterations before the current round, where the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; and use the truncation error to update the second predicted noise signal obtained in the current round of iteration.

[0018] In a possible implementation, before using the truncation error to update the second predicted noise signal obtained in the current round of iteration, the second processing module is further configured to: perform overcorrection on the truncation error.

[0019] In a possible implementation, during the iterative process of adding and removing noise from the first content, the amount of noise added is greater than the amount of noise removed.

[0020] In one possible implementation, before denoising the first predicted noise signal through the diffusion model iteration, the second processing module is further used to: filter out a subsequence of a first length from the inference sequence of the diffusion model, where the subsequence is used to represent the number of times the first predicted noise signal is denoised and the degree of denoising each time.

[0021] In a possible implementation, the number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

[0022] In one possible implementation, the content generation device may be configured on a server. In this case, the content generation device may further include a communication module. The communication module is configured to obtain first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling using a diffusion model; and to transmit the second content related to the first data to the electronic device after the second processing module obtains the second content.

[0023] In a third aspect, the present application provides a content generation device, comprising a processor and a memory, wherein the memory is configured to store computer instructions. When the computer instructions are executed by the processor, the content generation device performs the method provided in the first aspect. Exemplarily, the content generation device may be, but is not limited to, a chip.

[0024] In a fourth aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0025] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation of the first aspect; alternatively, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0026] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation of the first aspect; or, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0027] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of the architecture of a content generation system provided in an embodiment of the present application;

[0029] FIG2 is a flow chart of a content generation method provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of a process for correcting denoising results provided by an embodiment of the present application;

[0031] FIG4 is a schematic structural diagram of a content generation device provided in an embodiment of the present application;

[0032] FIG5 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0033] FIG6 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0034] FIG7 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0036] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0037] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0038] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0039] For example, Figure 1 shows a schematic diagram of the architecture of a content generation system provided by an embodiment of the present application. As shown in Figure 1, the content generation system 100 may include: a client 110 and a server 120. The client 110 and the server 120 may communicate with each other via a network, but is not limited to a network.

[0040] Among them, the client 110 is mainly responsible for receiving data input by the user (such as text, voice, pictures, etc.), and transmitting the data to the server 220, as well as receiving content related to the data input by the user, such as pictures, videos, audio, etc., generated by the server 220. Among them, the client 110 can be a desktop application, a mobile application, a web application or a web-based application, etc. In this embodiment, the user can input data such as text on the client 110. After the user completes the input and triggers content generation on the client 110, the client 110 can send the data input by the user to the server 120. Exemplarily, after completing the input, the user can click a button on the client 110 for executing the content generation task. For example, when the client 110 is a web application, the user can click the "enter" button on the keyboard to trigger the content generation task.

[0041] The server 120 is primarily responsible for receiving data from the client 10 and, based on the received data, generating content related to the data and sending the content to the client 110. For example, a diffusion model 121 may be configured in the server 120. The server 120 can generate content related to the user input data using the diffusion model 121 configured thereon. Furthermore, the server 110 is typically deployed on a server and has certain computing and storage resources. For example, the server on which the server 120 is deployed may be, but is not limited to, a cloud server, a physical server, or a virtual private server.

[0042] The above is an introduction to the content generation system 100 provided in the embodiment of the present application. Next, based on the above content, a content generation method provided in the embodiment of the present application is introduced.

[0043] For example, FIG2 shows a flow chart of a content generation method provided by an embodiment of the present application. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. As shown in FIG2, the content generation method may include the following steps:

[0044] S201 : In response to first data obtained from a user input, an initial signal obtained by random sampling is processed by a diffusion model to obtain a first predicted noise signal.

[0045] In this embodiment, after obtaining first data input by the user, random sampling can be performed using a diffusion model to obtain an initial signal. For example, the first data can be text, an image, or audio. The sampled initial signal is then processed using the diffusion model to obtain a first predicted noise signal.

[0046] S202: De-noise the first predicted noise signal through diffusion model iteration to obtain first content related to the first data.

[0047] In this embodiment, after obtaining the first predicted noise signal, the first predicted noise signal can be denoised through iteration of the diffusion model to obtain first content related to the first data, such as images, audio, video, etc. Exemplarily, during the iteration of the diffusion model, the degree of denoising can vary in each round. Exemplarily, the denoising result of round t+1 can be:

[0048] Among them, x t+1 is the denoising result of the t+1th round, x t is the denoising result of the tth round, α and are all hyperparameters, ∈ θ (x t ,t) is the estimated noise of the tth round.

[0049] In some embodiments, in order to improve the accuracy of the denoising results, the results obtained in the current round of iterations can be updated using the results obtained in the previous rounds of iterations. Specifically, as shown in FIG3 , the following steps can be included:

[0050] S301: Determine a truncation error in an iterative process based on a predicted noise signal generated in k (k≥1) rounds of iterations before the current round, wherein the truncation error is used to characterize the deviation between an ideal denoising result and an actual denoising result.

[0051] In this embodiment, the value of k can be pre-set or randomly generated. In the current round of iteration, the prediction noise signal generated by the k rounds of iteration before the current round (i.e., a part of the historical prediction noise signal or the entire historical prediction noise signal) can be used to determine the truncation error in the iteration process. For example, the truncation error of the t+1 round can be determined by the following formula:

[0052] Among them, r(t,i) is the correction coefficient, x t-i The result generated by the ti-th round iteration, α and are all hyperparameters, ∈ θ (x t ,t) is the estimated noise of round t. Exemplarily, r(t,1) to r(t,k) can be a geometric progression.

[0053] S302: Using the truncation error, update the second predicted noise signal obtained in the current round of iteration.

[0054] In this embodiment, after obtaining the truncation error, the truncation error can be used to update the second predicted noise signal obtained in the current round of iteration. From the above "Formula 1" and "Formula 2", it can be obtained that the predicted noise signal generated in the t+1th round is:

[0055] After the first content is obtained through iteration, S203 may be executed.

[0056] S203: Perform noise addition and removal processing on the first content through diffusion model iteration to obtain second content related to the first data.

[0057] In this embodiment, after obtaining the first content, the diffusion model can be used to iteratively add and remove noise from the first content to obtain second content related to the first data. During the current iteration, noise can be randomly sampled and added to the content to be processed, followed by further denoising. By re-degrading the results obtained during the first iteration using random resampling and then further denoising, local details can be enhanced, restoring more detailed information and optimizing the quality of the resulting content.

[0058] In this way, through a two-stage content generation approach, the content structure (e.g., image structure, etc.) can be quickly constructed in the first stage with fewer inferences, and the content details can be improved in the second stage, thereby generating high-quality content more efficiently and making the structural details rich and realistic. Since local details can be improved in the second stage, only the content structure needs to be constructed in the first stage without constructing local details, so the number of iterations in the first stage can be reduced, and the reduced number of iterations can be less than the number of iterations in the second stage, which reduces the total number of iterations, thereby improving content generation efficiency while ensuring the quality of the generated content. At the same time, the quality of the generated content can also be improved while the total number of iterations remains unchanged.

[0059] In some embodiments, S202 can be understood as the process of restoring low-frequency information. The first content generated at this time is the overall structure and primary texture of the final content. S203 can be understood as the process of restoring high-frequency information. This involves adjusting the local texture, contrast, etc. of the content generated in S202 to enhance local details and optimize content quality.

[0060] For example, when the content required by the user is a face image, the first content obtained in S202 may be the shape of the face and facial features, as well as the skin texture of the face, and the second content obtained in S203 may be the result of adjusting the local texture and contrast of the first content.

[0061] When the content required by the user is an audio segment, the first content obtained through S202 may be a basic melody, and the second content obtained through S203 may be some drum beats and lyrics.

[0062] In some embodiments, in the aforementioned S302, the truncation error can be overcorrected. For example, if the original correction coefficient is a geometric progression, different correction coefficients can be multiplied by different values. In this way, overcorrection can prevent the result generated by the first stage from being too smooth, and can preserve the local primary texture, thereby providing more room for adjustment in the second stage, allowing the second stage to play a greater role and further improving the quality of content generation. In this case, in the second stage, the coefficient for adding noise can be greater than the coefficient for removing noise, where these two coefficients can be hyperparameters. In this way, in the second stage, the amount of noise added is greater than the amount of noise removed. It should be understood that since the first stage has already caused the result to converge to the target position (i.e., the optimal denoising result), adding more noise in the second stage can cause the result generated in the first stage to oscillate around the target position. During the oscillation process, the distance from a noise point (i.e., a noise point) to the target position is generally smaller than the distance between two noise points. Therefore, a smaller coefficient is used during denoising to remove a small amount of noise.

[0063] In some embodiments, the number of iterations in the second stage (i.e., S203) can be a hyperparameter, such as 2 or 3. The number of iterations in the first stage (i.e., S202) can also be a hyperparameter, such as 7 or 8. For example, since the first stage has already converged from a random position to near the target position, and in order to facilitate rapid fine-tuning in the second stage, the number of iterations in the second stage can be smaller than that in the first stage. This can improve inference speed and reduce resource overhead while ensuring the quality of generated content.

[0064] In some embodiments, the diffusion model may have a preset inference sequence, requiring different levels of noise removal at different inference steps. For example, when the inference sequence is 1 to 1000, noise removal should begin at step 1000, where the level of denoising may be the highest, and then decrease in sequence. Exemplarily, the numbers in the inference sequence can be understood as the number of inference steps. Furthermore, to speed up the inference of the diffusion model, before the first stage (i.e., S202), a subsequence of a first length can be filtered from the inference sequence of the diffusion model. This subsequence can be used to represent the number of denoising steps (i.e., the number of iterations) in the first stage, as well as the level of denoising achieved each time. Exemplarily, the subsequence can be uniformly sampled from the inference sequence. For example, when the inference sequence is 1 to 1000, if the first length is 10, the subsequences can be 1000, 900, 800, 700, 600, 500, 400, 300, 200, and 100.

[0065] It is understandable that the content generation method shown in FIG2 above can be executed by the terminal side (such as a computer or other electronic device) or by the cloud side (such as a server, cloud server, etc.). When executed by the terminal side, after the terminal side obtains the second content, the second content can be presented to the user. When executed by the cloud side, the terminal side can obtain the first data input by the user, and then transmit the first data to the cloud side. In this way, the cloud side can obtain the first data from the terminal side and process the first data; after obtaining the second content, the cloud side can transmit the second content to the terminal side so that the second content can be presented to the user on the terminal side.

[0066] It can be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0067] The above is an introduction to the content generation method provided by the embodiment of the present application. Next, based on the method in the above embodiment, a content generation device provided by the embodiment of the present application is introduced.

[0068] For example, FIG4 shows a schematic diagram of the structure of a content generation device provided by an embodiment of the present application. As shown in FIG4 , there are a first processing module 401 and a second processing module 402. The first processing module 401 is used to process the initial signal obtained by random sampling through a diffusion model in response to the first data input by the user, so as to obtain a first predicted noise signal. The second processing module 402 is used to denoise the first predicted noise signal through iterative diffusion model to obtain first content related to the first data; and to add and remove noise to the first content through iterative diffusion model to obtain second content related to the first data.

[0069] In some embodiments, when iteratively denoising the first predicted noise signal, the second processing module 402 is specifically used to: determine the truncation error in the iteration process based on the predicted noise signal generated in k rounds (k≥1) before the current round, where the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; and use the truncation error to update the second predicted noise signal obtained in the current round of iteration.

[0070] In some embodiments, before using the truncation error to update the second predicted noise signal obtained in the current round of iteration, the second processing module 402 is further configured to perform overcorrection on the truncation error.

[0071] In some embodiments, during the iterative process of adding and removing noise from the first content, the amount of noise added is greater than the amount of noise removed.

[0072] In some embodiments, before denoising the first predicted noise signal through the diffusion model iteration, the second processing module 402 is further used to: filter out a subsequence of a first length from the inference sequence of the diffusion model, where the subsequence is used to represent the number of times the first predicted noise signal is denoised and the degree of denoising each time.

[0073] In some embodiments, the number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

[0074] In some embodiments, the content generation device 400 is configured on a server. In this case, the content generation device 400 may further include a communication module (not shown). The communication module is configured to obtain first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling using a diffusion model; and to transmit the second content related to the first data to the electronic device after the second processing module obtains the second content.

[0075] In some embodiments, the first processing module 401 and the second processing module 402 shown in FIG4 can be implemented via software or hardware. For example, the implementation of the first processing module 401 will be described below using the first processing module 401 as an example. Similarly, the implementation of the second processing module 402 can refer to the implementation of the first processing module 401, and the implementation of the communication module (not shown) can also refer to the implementation of the first processing module 401.

[0076] As an example of a software functional unit, the first processing module 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first processing module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0077] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0078] As an example of a hardware functional unit, the first processing module 401 may include at least one computing device, such as a server. Alternatively, the first processing module 401 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0079] The multiple computing devices included in the first processing module 401 can be distributed in the same region or in different regions. The multiple computing devices included in the first processing module 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first processing module 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0080] It should be noted that, in other embodiments, the first processing module 401 can be used to execute any step in the content generation method described in the above embodiments, and the second processing module 402 can be used to execute any step in the content generation method described in the above embodiments. The steps that the first processing module 401 and the second processing module 402 are responsible for implementing can be specified as needed. The first processing module 401 and the second processing module 402 respectively implement different steps in the content generation method described in the above embodiments to achieve all the functions of the content generation device 400 shown in Figure 4.

[0081] This application also provides a computing device 500. As shown in Figure 5, computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. Computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 500.

[0082] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG5 shows a single line, but this does not imply a single bus or type of bus. Bus 504 may include a path for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, and communication interface 508).

[0083] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0084] The memory 506 may include volatile memory, such as random access memory (RAM). The processor 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0085] Memory 506 stores executable program code, which processor 504 executes to implement the functions of first processing module 401 and second processing module 402 shown in FIG. 4 , thereby implementing the content generation method described in the above embodiment. In other words, memory 506 stores instructions for executing the content generation method described in the above embodiment.

[0086] Alternatively, the memory 506 stores executable code, and the processor 504 executes the executable code to respectively implement the functions of the content generation device 400 shown in Figure 4, thereby implementing the content generation method described in the above embodiment. In other words, the memory 506 stores instructions for executing the content generation method described in the above embodiment.

[0087] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0088] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0089] As shown in Figure 6, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the content generation method described in the above embodiment.

[0090] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the content generation method described in the above embodiment. In other words, the combination of one or more computing devices 500 can jointly execute instructions for executing the content generation method described in the above embodiment.

[0091] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster may store different instructions, each for executing part of the functions of the content generation apparatus 400 shown in FIG4 . In other words, the instructions stored in the memory 506 in different computing devices 500 may implement the functions of one or more of the first processing module 401 and the second processing module 402.

[0092] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 7 illustrates a possible implementation. As shown in FIG. 7 , two computing devices 500A and 500B are connected via a network. Specifically, the connection to the network is achieved via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the first processing module 501. Simultaneously, the memory 506 in the computing device 500B stores instructions for executing the functions of the second processing module 502.

[0093] It should be understood that the functionality of the computing device 500A shown in FIG7 may also be implemented by multiple computing devices 500. Similarly, the functionality of the computing device 500B may also be implemented by multiple computing devices 500.

[0094] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between computing device clusters described in Figures 6 and 7. However, the memory 506 in one or more computing devices 500 in this computing device cluster can store the same instructions for executing the method described in the above embodiment.

[0095] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store some instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for executing the aforementioned data processing method.

[0096] Based on the methods in the above embodiments, embodiments of the present application provide a content generation device, including a processor and a memory, wherein the memory is configured to store computer instructions. When the computer instructions are executed by the processor, the content generation device 400 shown in FIG4 executes the methods in the above embodiments. Exemplarily, the content generation device may be, but is not limited to, a chip.

[0097] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc.

[0098] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment, or, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above embodiment.

[0099] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0100] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0102] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A content generation method, characterized in that: include: In response to the acquired first data input by the user, an initial signal obtained by random sampling is processed by a diffusion model to obtain a first predicted noise signal; Iteratively denoising the first predicted noise signal through the diffusion model to obtain first content related to the first data; The first content is iteratively processed by adding and removing noise through the diffusion model to obtain second content related to the first data.

2. The method according to claim 1, characterized in that Iteratively denoising the first prediction noise signal, comprising: In the current round of iteration, based on the predicted noise signal generated in k rounds (k≥1) of iteration before the current round, a truncation error in the iteration is determined, wherein the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; The second predicted noise signal obtained in the current round of iteration is updated using the truncation error.

3. The method according to claim 2, characterized in that Before using the truncation error to update the second predicted noise signal obtained in the current round of iteration, the method further includes: The truncation error is overcorrected.

4. The method according to claim 3, characterized in that In the process of iteratively performing noise adding and removing processing on the first content, the amount of noise added is greater than the amount of noise removed.

5. The method according to any one of claims 1 to 4, characterized in that: Before denoising the first predicted noise signal through the diffusion model iteration, the method further includes: A subsequence of a first length is selected from the inference sequence of the diffusion model, and the subsequence is used to represent the number of times the first prediction noise signal is denoised and the degree of denoising each time.

6. The method according to any one of claims 1 to 5, characterized in that: The number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to a server; Before the initial signal obtained by random sampling is processed by the diffusion model, it also includes: Acquire the first data sent by the electronic device; After obtaining the second content related to the first data, the method further includes: The second content is sent to the electronic device.

8. A content generating device, characterized in that: include: A first processing module, configured to process an initial signal obtained by random sampling through a diffusion model in response to first data input by a user, to obtain a first predicted noise signal; a second processing module, configured to perform denoising on the first prediction noise signal through iteration of the diffusion model to obtain first content related to the first data; The second processing module is further configured to perform noise addition and removal processing on the first content through iteration of the diffusion model to obtain second content related to the first data.

9. The device according to claim 8, characterized in that When the second processing module iteratively denoises the first predicted noise signal, it is specifically used to: In the current round of iteration, based on the predicted noise signal generated in k rounds (k≥1) of iteration before the current round, a truncation error in the iteration is determined, wherein the truncation error is used to characterize the deviation between the ideal denoising result and the actual denoising result; The second predicted noise signal obtained in the current round of iteration is updated using the truncation error.

10. The device according to claim 9, characterized in that Before the second processing module updates the second predicted noise signal obtained by the current round of iteration using the truncation error, it is further configured to: The truncation error is overcorrected.

11. The device according to claim 10, characterized in that In the process of iteratively performing noise adding and removing processing on the first content, the amount of noise added is greater than the amount of noise removed.

12. The device according to any one of claims 8 to 11, characterized in that: Before denoising the first predicted noise signal through the diffusion model iteration, the second processing module is further used to: A subsequence of a first length is selected from the inference sequence of the diffusion model, and the subsequence is used to represent the number of times the first prediction noise signal is denoised and the degree of denoising each time.

13. The device according to any one of claims 8 to 12, characterized in that: The number of iterations for obtaining the first content is greater than the number of iterations for obtaining the second content.

14. The device according to any one of claims 8 to 13, characterized in that: The device is configured on a server, and the device further comprises: a communication module; The communication module is used to obtain the first data sent by the electronic device before the first processing module processes the initial signal obtained by random sampling through the diffusion model; and after the second processing module obtains the second content related to the first data, send the second content to the electronic device.

15. A content generating device, characterized in that: The device comprises a processor and a memory, wherein the memory is used to store computer instructions. When the computer instructions are executed by the processor, the content generating device executes the method according to any one of claims 1 to 7.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the instructions are executed by a computing device cluster, the computing device cluster causes the computing device cluster to perform the method according to any one of claims 1 to 7, wherein the computing device cluster comprises at least one computing device.

18. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster includes at least one computing device.

Images