Control method and device and electronic equipment

Abstract

The embodiment of the invention provides a control method and device, and electronic equipment, which are used for improving the robustness or operation efficiency of a diffusion model. The method is used for controlling operation of a diffusion model and comprises the steps that a first sampling time step for executing the diffusion model is controlled, and a first randomness parameter is adopted in the first sampling time step; controlling and executing a second sampling time step of the diffusion model, wherein the second sampling time step adopts a second randomness parameter; wherein the randomness introduced by the first randomness parameter in the first sampling time step is greater than the randomness introduced by the second randomness parameter in the second sampling time step, and the first sampling time step is earlier than the second sampling time step. Therefore, the robustness or operation efficiency of the diffusion model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more particularly to a control method and apparatus, as well as an electronic device. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI models (such as neural network models, Transformer models, or generative models) are playing an increasingly important role in various intelligent applications. Diffusion models, as a powerful generative model, transform random noise into generated content through a reverse denoising process. Improving the robustness and operational efficiency of diffusion models in applications has become a pressing issue. Summary of the Invention

[0003] This application provides a control method and apparatus, as well as an electronic device, for improving the robustness or operational efficiency of a diffusion model.

[0004] In a first aspect, a control method is provided for controlling the operation of a diffusion model. The control method includes: controlling the execution of a first sampling time step of the diffusion model, wherein the first sampling time step adopts a first randomness parameter; controlling the execution of a second sampling time step of the diffusion model, wherein the second sampling time step adopts a second randomness parameter; wherein the randomness introduced by the first randomness parameter in the first sampling time step is greater than the randomness introduced by the second randomness parameter in the second sampling time step, and the first sampling time step is earlier than the second sampling time step.

[0005] Based on the above technical solution, by introducing greater randomness through a first randomness parameter at a relatively early first sampling time step, the directionality of the accumulated error caused by noise in the relatively early sampling time step is reduced, thereby improving the robustness of the diffusion model. Furthermore, by employing a second randomness parameter at the second sampling time step, which is temporally located after the first sampling time step, less randomness or diversity is introduced compared to the first randomness parameter. This helps reduce the number of subsequent sampling time steps. This reduction in the number of time steps not only improves the running efficiency of the diffusion model but also enhances its robustness.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the diffusion model includes multiple sampling time steps, the first N sampling time steps of the multiple sampling time steps include the first sampling time step, and the last M sampling time steps of the multiple sampling time steps include the second sampling time step, where M and N are both positive integers.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the randomness parameter used in the first N sampling time steps belongs to the first value range, and the randomness parameter used in the last M sampling time steps takes the value of 0, and the minimum value in the first value range is greater than 0.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the proportion of the first N sampling time steps in multiple sampling time steps does not exceed 50%.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first N sampling time steps account for 30% to 40% of the total sampling time steps.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the randomness parameters of the N sampling time steps are the same or different.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the value range of the first randomness parameter is 0.4 to 1.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the number of sampling time steps of the diffusion model is less than or equal to the first threshold.

[0013] In a second aspect, a control device is provided, comprising at least one processing circuit and a storage circuit, the processing circuit being configured to execute a control method as described in any of the first aspects, and the storage circuit being configured to store data for use by the processing circuit.

[0014] Thirdly, an electronic device is provided, including the control device as described in the second aspect. Attached Figure Description

[0015] Figure 1 A schematic flowchart of a control method according to an exemplary embodiment of this application is shown;

[0016] Figure 2 This diagram illustrates the performance of DDPM trained on the CIFAR-10 dataset under different noise conditions.

[0017] Figure 3 This diagram illustrates the performance of DDIM trained on the CIFAR-10 dataset under different noise conditions.

[0018] Figure 4 This diagram illustrates a comparison of image generation quality when noise is introduced at different stages of the sampling process in a diffusion model.

[0019] Figure 5 A schematic diagram comparing the accumulated error and the intra-step error in the inverse denoising process of the diffusion model is shown. Figure 6A schematic diagram of an in-memory computing system according to an exemplary embodiment of this application is shown;

[0020] Figure 7 A schematic diagram of another in-memory computing system according to an exemplary embodiment of this application is shown;

[0021] Figure 8 A schematic diagram comparing the sampling methods based on traditional DDPM and DDIM with the control method of this application is shown;

[0022] Figure 9 The diagram illustrates the performance of the diffusion models corresponding to the control method provided in this application and three other sampling methods on different noise environments and datasets.

[0023] Figure 10 The diagram illustrates ADC noise, programming noise, and nonlinear noise.

[0024] Figure 11 The performance comparison of the diffusion models corresponding to the control method of this application and three other sampling methods is shown in three different hardware noise environments.

[0025] Figure 12 This application illustrates a control device provided in this application;

[0026] Figure 13 A schematic diagram of an electronic device according to an exemplary embodiment of this application is shown. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0028] To keep the drawings concise, the figures in this application only schematically show the parts related to the corresponding embodiments, and they do not represent the actual structure of the product. In addition, to make the drawings concise and easy to understand, some figures only schematically show some structures or components, and there may actually be more or fewer identical or similar structures or components.

[0029] The business scenarios described in the embodiments of this application are for illustrative purposes only and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0030] In this application, unless otherwise expressly specified and limited, "connection" includes direct or indirect connection between objects: connected objects may be directly connected through a medium (e.g., wires, traces, etc.), or indirectly connected through other components, or may be an internal connection. "Coupling" includes signal connection between objects, which may be achieved directly through a medium (e.g., wires, traces, etc.), or through other components. "Grounding" includes direct grounding or indirect grounding, with indirect grounding including, for example, grounding through other components.

[0031] In this application, unless otherwise expressly specified and limited, ordinal numbers, such as "first," "second," etc., are used only to distinguish the objects being described and should not be construed as indicating or implying the relative importance or order between the objects being described. Furthermore, ordinal numbers do not represent the quantity of the objects being described. "Multiple" includes two or more, and other quantifiers are similar. "Or," "and / or," etc., are used to describe the relationship between objects, indicating a non-exclusive inclusion. For example, "A and / or B," "A or B" can include: "A alone," "B alone," or "A and B." Similarly, "A, B, and / or C," "A, B, or C" can include: "A alone," "B alone," "C alone," "A and B," "A and C," "B and C," or "A, B, and C." Additionally, the " / " in this application is used to indicate an "or" relationship between preceding and following objects. The meaning of "one or more of A and B" or "at least one of A and B" in this application is the same as the meaning of "A and / or B" or "A or B" above. "One or more of A, B and C" or "at least one of A, B and C" has the same meaning as "A, B and / or C" or "A, B or C" above.

[0032] In the field of generative artificial intelligence (AIGC), the use of AI algorithms to generate content is gaining increasing attention. Diffusion models, as a powerful generative model, add noise to the original data through forward diffusion and then gradually remove the noise through reverse diffusion, obtaining the original data or data with a similar distribution. A trained diffusion model can use the reverse diffusion denoising process to gradually remove noise and obtain the generated content. Diffusion models can be used to generate images, audio, video, and more. In applications, diffusion models can be deployed on hardware devices, utilizing hardware resources for training and / or inference.

[0033] The back-diffusion process of a diffusion model can include a sampling process, which includes multiple sampling time steps. Extensive research by the applicant has found that introducing higher randomness or diversity into relatively early sampling time steps can improve the robustness of the diffusion model. Therefore, embodiments of this application provide a control method for controlling the operation of a diffusion model. This control method increases the robustness of the diffusion model by adjusting the randomness or diversity of the sampling time steps during the back-diffusion (or back-denoising) process. For example, introducing higher randomness or diversity into the sampling time steps in relatively early stages of the back-diffusion process can reduce directional cumulative error and improve the robustness of the diffusion model.

[0034] For example, Figure 1 A schematic flowchart of a control method according to an exemplary embodiment of this application is shown. Figure 1 As shown, the control method 100 may include the following steps:

[0035] Step 101: Control the execution of the first sampling time step of the diffusion model. The first sampling time step adopts the first randomness parameter.

[0036] Step 102: Control the execution of the second sampling time step of the diffusion model. The second sampling time step adopts a second randomness parameter. The randomness introduced by the first randomness parameter in the first sampling time step is greater than the randomness introduced by the second randomness parameter in the second sampling time step. The first sampling time step is earlier than the second sampling time step.

[0037] The above randomness parameters can also be called diversity parameters, which are mainly related to the randomness or diversity of content generation.

[0038] The applicant's research revealed that the diffusion model is vulnerable to noise. Further investigation into this vulnerability revealed that instantaneous errors introduced by noise in relatively early sampling time steps can create directional cumulative errors in subsequent inverse denoising processes. This directional cumulative error significantly impacts the diffusion model's inference process; for example, in image generation, this error manifests as all pixels collapsing towards one or two color channels. This application's embodiments mitigate this directionality by adjusting the randomness or diversity of the sampling time steps during inference. For instance, in this application's embodiments, introducing greater randomness or diversity through a first randomness parameter in a relatively early first sampling time step reduces the directionality of the cumulative error caused by noise in those earlier sampling time steps, thereby improving the robustness of the diffusion model. Furthermore, the second sampling time step, which is located after the first sampling time step in terms of time sequence, adopts a second randomness parameter. Compared with the first randomness parameter, it introduces less randomness or diversity, which is beneficial to reduce the number of subsequent sampling time steps. Moreover, the applicant's research found that reducing the number of time steps can not only improve the running efficiency of the diffusion model, but also improve the robustness of the diffusion model.

[0039] As an example, taking images generated by the denoising diffusion probabilistic model (DDPM) and the denoising diffusion implicit model (DDIP) as examples, the vulnerability of diffusion models under noise conditions was studied from the perspective of the reverse denoising process. Figure 2 This diagram illustrates the performance of DDPM trained on the CIFAR-10 dataset under different noise conditions. Figure 3 The diagram illustrates the performance of DDIM trained on the CIFAR-10 dataset under different noise conditions. Figure 2 and Figure 3 The vertical axis represents the Fre'chet inception distance (FID) score, and the horizontal axis represents the noise standard deviation. For example, noise standard deviations of 0.00, 0.02, 0.04, 0.06, 0.08, and 0.10 indicate noise. A noise standard deviation of 0.00 indicates no added noise. The FID score is negatively correlated with the image generation quality. A higher FID score indicates lower image generation quality, and a lower FID score indicates higher image generation quality.

[0040] like Figure 2 As shown, the test objects are DDPMs with 100 sampling time steps, 500 sampling time steps, and 1000 sampling time steps, respectively. Tests were conducted under different noise standard deviations, and the FID scores of the DDPMs with different sampling time steps under these different noise standard deviations were obtained. (The last sentence appears to be incomplete and possibly refers to a separate topic.) Figure 2 For any fixed number of sampling time steps shown, the FID score of a DDPM increases with increasing noise standard deviation, indicating a decrease in image generation quality. Under noise conditions with a large noise standard deviation (e.g., 0.04–0.10), the FID score of a DDPM with fewer sampling time steps is lower than that of a DDPM with more sampling time steps.

[0041] like Figure 3 As shown, the test objects are DDIMs with 20 sampling time steps, 50 sampling time steps, and 100 sampling time steps, respectively. Tests were conducted under different noise standard deviations, and the FID scores of DDIMs with different sampling time steps under different noise standard deviations were obtained. For... Figure 3 For the DDIM with any fixed number of sampling time steps shown, under noise conditions with a small noise standard deviation (e.g., 0–0.04), the FID score of the DDIM decreases as the noise standard deviation increases, indicating an improvement in image generation quality. Conversely, under noise conditions with a large noise standard deviation (e.g., 0.04–0.10), the FID score of the DDIM increases as the noise standard deviation increases, indicating a decrease in image generation quality. Under noise conditions with a large noise standard deviation (e.g., 0.04–0.10), the FID score of the DDIM with fewer sampling time steps is lower than that of the DDIM with more sampling time steps.

[0042] based on Figure 2 and Figure 3 It can be seen that DDPM and DDIM exhibit performance degradation under significant hardware noise. For example, the DDPM model with 1000 sampling time steps shows an FID score exceeding 100 under Gaussian noise with a standard deviation of 0.10 compared to the FID score under noise-free conditions, demonstrating the vulnerability of diffusion models to noise. Furthermore, based on... Figure 2 and Figure 3It can also be seen that reducing the sampling time step can bring better robustness to the diffusion model. The applicant also used other datasets for research and testing. For example, comparing face images generated by DDIM trained on the Celeb A dataset with 100 sampling time steps under different noise standard deviations revealed that DDIM generates high-quality images with clear facial features, clear skin texture, and well-defined details under noise conditions. However, as the noise standard deviation is continuously increased under noisy conditions, the images generated by DDIM exhibit blurred features, reduced color fidelity, and loss of fine details, demonstrating performance degradation. This shows that the diffusion model is vulnerable to noise conditions.

[0043] Further research by the applicant on the vulnerability of the above diffusion model revealed that, during the reverse denoising process of the diffusion model, the instantaneous error caused by noise introduced in relatively early sampling time steps can accumulate into a significant error in subsequent reverse denoising processes. This is one of the main reasons for the vulnerability of the diffusion model. For example, error propagation during the reverse denoising process can be studied and analyzed according to some embodiments. Figure 4 This diagram illustrates a comparison of image generation quality when noise is introduced at different stages of the sampling process (reverse denoising process) in the diffusion model. Figure 4 The vertical axis represents the FID score, and the horizontal axis represents the sampling time step range where noise is introduced. "S1-S2" indicates that noise is applied from sampling time step S1 to sampling time step S2; for example, "80-60" indicates that noise is applied from sampling time step 80 to sampling time step 60. Repeated boundary points can belong to the previous range, the next range, or both ranges. Figure 4 It can be seen that the timing of noise injection (or the range of sampling time steps) significantly affects the final image generation quality, which can be measured by the FID score. When noise is applied in the relatively early stages of the reverse denoising process (sampling time steps ranging from 99 to 80), the FID score is higher, but as noise injection moves to later stages, the FID score drops sharply. Furthermore, the applicant confirmed these quantitative findings using visual samples corresponding to different stages of the reverse denoising process, indicating that images generated by later noise injection have slight artifacts, while noise in relatively early stages leads to severe distortion, color shifts, or structural damage. This time sensitivity stems from the accumulation of errors during iterative sampling.

[0044] Figure 5 A schematic diagram comparing the cumulative error and intra-step error in the diffusion model during the inverse denoising process is shown. Figure 5The horizontal axis represents the sampling time step, and the vertical axis represents the cumulative error introduced by noise, which can be represented, for example, by the root mean square error (RMSE). Figure 5 As shown, the cumulative error (curve 1) increases exponentially during the reverse denoising process, while the intra-step error (curve 2) remains almost constant at around 1. This demonstrates that errors introduced in relatively early sampling time steps propagate and amplify through subsequent iterations. The applicant further confirmed this using visual samples corresponding to different sampling time steps. The exponential growth of the cumulative error during reverse denoising indicates that reducing the impact of relatively early noise is more effective in improving image generation quality.

[0045] As can be seen from the study of the vulnerability of the diffusion model, introducing noise in a relatively early sampling time step not only produces an immediate error at the sampling time step where the noise is introduced, but also accumulates the error in subsequent sampling time steps, thus causing a decrease in the quality of image generation due to the accumulation of errors throughout the sampling process.

[0046] The aforementioned noise is introduced during the operation of the diffusion model due to its operating environment and accumulates during the reverse diffusion process, affecting the quality of content generation. This application's embodiments introduce greater randomness or diversity by using a first randomness parameter at a relatively early first sampling time step. This increases the randomness or diversity of the relatively early sampling time step, thereby reducing the directionality of the accumulated error caused by noise at the relatively early sampling time step, and thus improving the robustness of the diffusion model.

[0047] The diffusion model can be deployed on hardware resources. The noise mentioned above can include hardware noise, such as device-level differences caused by manufacturing, environment, or aging, and circuit-level noise caused by voltage drop and leakage current.

[0048] In some embodiments, some or all of the computations in the diffusion model can be run using an in-memory computing architecture. In other embodiments, some or all of the computations in the diffusion model can also be run using other architectures, such as the von Neumann architecture. Alternatively, the diffusion model can be run using a hybrid architecture of in-memory computing and the von Neumann architecture. This application does not impose further limitations on these embodiments.

[0049] In traditional computing paradigms, such as the von Neumann architecture, storage and computation are physically separated. When processing data using this paradigm, data is frequently transferred between storage devices and computing devices, resulting in data transfer latency and energy consumption. Utilizing an in-memory computing architecture to run part or all of the computation of a scaling model can further improve the efficiency of the scaling model and reduce its energy consumption during operation.

[0050] In-memory computing (IMC) architecture can address the latency and energy consumption issues caused by frequent data movement between computing and storage devices, serving as a solution for optimizing the operational efficiency and energy efficiency of diffusion models. IMC architecture physically integrates storage and computing. This physical integration can include, for example, close integration of storage and computing components through packaging processes (known as near-memory computing); integrating processing circuitry within memory to achieve in-memory processing integration; or implementing computing through storage devices (known as in-memory computing), achieving tight integration of storage and computing. IMC architecture reduces data transfer requirements, lowers transmission latency and energy consumption, and significantly improves data processing efficiency.

[0051] This application provides a storage system. The storage system may include a storage circuit for storing data. The storage system may also include a readout circuit for converting and outputting the output signal of the storage circuit, thereby enabling the reading of the data stored in the storage circuit. The storage system may also include a control device for controlling the operation of the storage circuit, such as one or more operations including but not limited to programming (e.g., data writing), erasing, and reading (e.g., data retrieval). The control device may also control the operation of the readout circuit, such as enabling the readout circuit to convert its output signal, or controlling the timing of multiple modules in the readout circuit.

[0052] In some possible embodiments, the storage system can be used in a memory-computing architecture, also known as a memory-computing system. The control device can be used to control one or more operations of the storage circuit, such as programming, erasing, computation, and reading (including reading data or computation results). The reading operation can include reading stored data and / or reading (or sensing) computation results; that is, the output of the storage circuit can include the output of stored data or the output of computation results. Optionally, the control device can be integrated with the storage circuit and call up data stored in the storage circuit, and perform computations based on the called data. Optionally, the control device can control the in-memory computation of the storage circuit and control the reading of the computation results of the storage circuit. Optionally, the control device can also process the output of the storage circuit.

[0053] This application does not limit the type of storage medium used in the storage circuit. The storage medium may include, but is not limited to, non-volatile memory (NVM) or volatile memory (VM). Volatile memory may include, but is not limited to, static random access memory (SRAM) or dynamic random access memory (DRAM). Non-volatile memory may include, but is not limited to, flash memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric memory (FeRAM), or phase change memory (PCM).

[0054] This application does not limit the structure of the control device. The control device may include one or more processing circuits (or processors). For example, the control device includes a processing circuit for controlling the operation of the storage circuit; the processing circuit may also control the operation of the read circuit. Alternatively, the control device may include a processing circuit for controlling the operation of the storage circuit, and another processing circuit for controlling the operation of the read circuit. Optionally, the control device may also include a post-processing circuit for post-processing the output of the storage circuit, such as decoding. Optionally, the control device may also include yet another processing circuit for processing the output of the storage circuit. This application does not limit the type of processing, and may include, but is not limited to, arithmetic operation processing, logic and control processing, data transmission processing, or multimedia data processing. The output of the storage circuit may include, for example, the output of data stored in the storage circuit or the output of the calculation result of in-memory calculation performed by the storage circuit.

[0055] For ease of understanding, Figure 6 A schematic diagram of a storage system according to an exemplary embodiment of this application is shown. Figure 1As shown, the storage system 600 may include a storage circuit 610, a control device 620, and a readout circuit 630. The storage circuit 610 can be used to store data; the control device 620 can be used to control the operation of the storage circuit 610, such as controlling one or more operations like programming, erasing, reading, or calculation; the readout circuit 630 is used to convert and output the output signal S of the storage circuit 610. The storage system 100 can be used in a memory-computing architecture; for example, the storage system 100 can support near-memory computation and / or in-memory computation. In near-memory computation, the control device 620 can be integrated with the storage circuit 610 and the readout circuit 630. During computation, the control device 620 controls the storage circuit 610 and the readout circuit 630 so that the storage circuit 610 outputs the stored data through the readout circuit 630; the control device 620 integrates a processing circuit for performing calculations on the data output by the storage circuit 610; optionally, the control device 620 can cache the data and control the processing circuit to retrieve the cached data for computation. In the in-memory calculation, the storage circuit 610 stores the weight data, and the control device 620 controls the storage circuit 610 to perform in-memory calculation on the input data based on the stored weight data. The control device 620 can also control the reading of the calculation results. Optionally, the control device 620 can also perform post-processing on the calculation results.

[0056] The output of the storage circuit 610 may include the output of stored data or the output of calculation results. The control device 620's control of the output of the storage circuit 610 may include the control of the following processes: the establishment of the output signal S and at least one conversion of the output signal S; the at least one conversion may include one or more of the following, such as signal type conversion, signal magnitude conversion, current-to-voltage conversion, analog-to-digital conversion, amplification, etc. For example, the control device 620 may control the readout circuit 630 to perform a first conversion and a second conversion on the output signal S; the first conversion includes current-to-voltage conversion, for example, the output signal S of the storage circuit 610 includes a current signal, and the readout circuit 630 can convert the current signal into a voltage signal; the second conversion includes analog-to-digital conversion, for example, the readout circuit 630 can convert the output of the first conversion into a digital signal. During the first conversion and / or the second conversion, the output signal may be amplified, or the output signal may be amplified after the first conversion or the second conversion; the readout circuit 630, which has the function of amplifying the output signal, may also be called an inductive amplifier circuit.

[0057] In some possible embodiments, the storage circuitry 610 may include one or more storage arrays. For ease of understanding, Figure 7 A schematic diagram of another storage system according to an exemplary embodiment of this application is shown.

[0058] like Figure 7As shown, the storage system 700 includes a storage circuit 710, which may include one or more storage arrays, such as storage arrays A1, A2, etc. Each storage array may include multiple storage cells. For example, storage array A1 is described here; other storage arrays may also be included in the storage circuit 710, and these other storage arrays are similar. Figure 7 As shown, the storage array A1 includes multiple storage cells S 11 -S mn Where m is the number of rows in storage array A1, and n is the number of columns in storage array A1. Storage cell S ij It can store data W ij Where i∈[1,m], j∈[1,n]. The storage system 700 may further include a control device 720 for controlling the operating state of the storage circuit 710. The control device 720 can control the storage array A1 to be in a programming state to send signals to the storage cells S. ij Write data W ij The control device 720 can control the storage array A1 to be in a data read-out state, so as to put the storage cell S... ij Stored data W ij Read out. Optionally, the storage array A1 can be used for in-memory computation, with storage unit S ij The conduction capability can be based on the stored data W ij And the data W changes. ij This can be referred to as weight data; the control device 720 can control the storage array A1 to be in a calculation state, and the control device 720 can send data to the storage cell S through the input terminal IN of the storage array A1. ij Provide input signal V i The input signal V i Acting on storage unit S ij Storage unit S ij The stored weight data W ij This makes the storage unit S ij It has a corresponding conduction capability, under which current flows out of or into the output terminal OUT. Multiple memory cells (e.g., S...) 1j -S mj The output terminals of the memory can be collinear. According to Kirchhoff's laws, the output signals of multiple memory cells are accumulated to obtain the output signal I. j The following formula (1) must be satisfied:

[0059]

[0060] As can be seen, a storage array can be used to perform multiplication and accumulation calculations, and the calculation result can be output at the output terminal. The sensing output of this calculation result can be understood as reading the calculation result. The multiple storage units with collinear outputs can be called a calculation group. The control device 700 can also control the storage circuit to be in the calculation result reading state to read the calculation result.

[0061] The storage system 700 may further include a readout circuit 730 for converting and outputting the signal output from the storage circuit 710. The signal output from the storage circuit 710 may characterize the data stored in the storage cell or the calculation result of the computing group. In some possible embodiments, the readout circuit 730 may include a first conversion circuit 731 and a second conversion circuit 732. The first conversion circuit 731 performs a first conversion on the output signal of the storage circuit 710 (e.g., readout current), for example, converting the current signal into a voltage signal; the second conversion circuit 732 performs a second conversion on the output of the first conversion, for example, converting the analog signal into a digital signal. The first conversion circuit 731 may also be called a sampling circuit. For example, the sampling circuit may be used to sample the current signal output by the storage circuit into a voltage signal. The second conversion circuit 732 may also be called a decision circuit. For example, the decision circuit may be used to convert the waveform parameters of the analog signal (e.g., voltage signal) into digital signals, such as converting the amplitude, pulse width, or area of ​​the analog signal into digital signals through decision. This application does not limit the accuracy of the conversion or decision, and may include 1-bit or multi-bit conversion accuracy. The control device 720 can control the operation of the readout circuit 730.

[0062] In some possible embodiments, different memory arrays can reuse the first conversion circuit 731 and / or the second conversion circuit 732; and / or, within the same memory array, different output terminals can reuse sub-circuits in the first conversion circuit 731 and / or the second conversion circuit 732 to reduce the hardware cost of the readout circuit and its footprint on the chip. In some possible embodiments, the control device 720 can control the operating timing of the readout circuit 730. By controlling the operating timing, the hardware resources of the readout circuit 730 can be reused by multiple memory arrays or multiple output terminals of a memory array to reduce the hardware overhead of peripheral circuits. The reused hardware resources may include, for example, the first conversion circuit 731, sub-circuits of the first conversion circuit 731, the second conversion circuit 732, or sub-circuits of the second conversion circuit 732. For example, a separate first sub-conversion circuit and / or second sub-conversion circuit can be set for one output terminal OUT of the storage circuit 710 to improve the read bandwidth. Taking the storage array A1 as an example, the first conversion circuit 221 may include multiple first conversion sub-circuits, which are respectively connected to multiple output terminals OUT of the storage array A1, and are used to perform a first conversion on the signal of the corresponding output terminal OUT. The second conversion circuit 732 may include multiple second conversion sub-circuits, which are respectively connected to multiple first conversion sub-circuits, and are used to perform a second conversion on the output of the first conversion sub-circuit. For example, multiple output terminals OUT of the storage circuit 710 can share the first conversion circuit or the first conversion sub-circuit. Optionally, multiple first conversion circuits or first conversion sub-circuits can share the second conversion circuit or the second conversion sub-circuit. Taking the storage array A1 as an example, the first conversion circuit 731 may include multiple first conversion sub-circuits, and one first conversion sub-circuit is connected to multiple output terminals OUT, and is used to perform a first conversion on the signals of the multiple output terminals in a time-division manner. Optionally, multiple first conversion sub-circuits can share the second conversion circuit 732, and the second conversion circuit 732 can perform time-division conversion on the output of the multiple first conversion sub-circuits. Optionally, the second conversion circuit 732 may include multiple second conversion sub-circuits, and one second conversion sub-circuit may be connected to multiple first conversion sub-circuits for time-division multiplexing of the outputs of the multiple first conversion sub-circuits. For ease of illustration, the output terminal of memory array A1 is used as an example above; the output terminals of other memory arrays or multiple memory arrays may adopt any of the above multiplexing methods. Optionally, multiple memory arrays may multiplex the first conversion circuit / first conversion sub-circuit and / or the second conversion circuit / second conversion sub-circuit. This application does not limit the number of first conversion circuits / first conversion sub-circuits and the number of second conversion circuits / second conversion sub-circuits; or, it does not limit the number of multiplexed output terminals of the first conversion circuit / first conversion sub-circuit and does not limit the number of multiplexed first conversion circuits / first conversion sub-circuits.

[0063] In some possible embodiments, the aforementioned storage unit may include a semiconductor device, and utilize the conductivity of the semiconductor device, such as electrical conductance or transconductance, to store data. For example, the storage unit may include a resistive storage device or a transistor storage device. For example, data storage can be achieved by controlling the conductivity of the resistive storage device, or by controlling the transconductance of the transistor storage device. Alternatively, the storage unit may utilize the energy stored in an energy storage element to store data, such as by utilizing the charge stored in a capacitor; this energy storage element may be connected to the semiconductor device, and the stored energy may act on the semiconductor device, causing the semiconductor device to generate a corresponding conductivity.

[0064] Figure 7 This is just an example illustrating how storage cells are connected in a storage array, except... Figure 7 Besides the connection method shown, other connection methods can also be used. For example, the input terminals of the memory cells can be connected in columns along a common line, and the output terminals of the memory cells can be connected in rows along a common line. Furthermore, the input terminals of the memory cells can include the gate of a transistor memory device, or the input terminals of the memory cells can include the source or drain of a transistor memory device; this application does not limit this. This application also does not limit the type of memory cell; for example, the memory cell can include, but is not limited to, transistors, memristors, magnetic tunnel junctions (MTJs), or phase-change structures. This application also does not limit the type of transistor, including, for example, metal-oxide-semiconductor field-effect transistors (MOSFETs), floating-gate transistors (FGTs), ferroelectric field-effect transistors (FeFETs), or thin-film transistors.

[0065] In some possible embodiments, the memory cell may include multiple transistors; for example, the memory cell may include a first transistor and a second transistor, wherein the driving terminal (e.g., gate) of the first transistor (which may be referred to as a "read transistor" or "read tube") and the non-driving terminal (e.g., source or drain) of the second transistor are connected, and the charge stored at the driving terminal of the first transistor or the voltage at the driving terminal of the first transistor can be used to characterize the stored data. Optionally, the memory cell may also include a capacitor, and the driving terminal of the first transistor may also be connected to the capacitor to increase the stability and duration of the stored charge or the voltage at the driving terminal.

[0066] In some embodiments of this application, the operation of the diffusion model can be accelerated using an in-memory computing architecture. In an in-memory computing architecture, the diffusion model is susceptible to hardware noise, such as device-level variations caused by manufacturing defects, environmental factors, or aging, and circuit-level noise caused by voltage drops and leakage current. When studying the vulnerability of the diffusion model to noise conditions, a noise model can be further constructed based on the noise characteristics of the in-memory chip for noise enhancement, and the vulnerability of the noise-enhanced diffusion model can then be studied.

[0067] For in-memory computing, the operation of the diffusion model includes computation in the analog domain. This computation is inevitably affected by hardware noise, such as the inherent non-ideals of semiconductor devices, which can interfere with the weight data of the diffusion model, causing fluctuations and reducing its reliability. The technical solution of this application has a more significant effect on in-memory computing architectures. For scenarios using in-memory computing to accelerate AI models, it can further improve the robustness of the diffusion model and reduce the impact of hardware noise. Moreover, using an in-memory computing architecture to accelerate the operation of the diffusion model can further improve its operating efficiency and save power consumption.

[0068] For example, the diffusion model in steps 101 and 102 above may include a trained DDPM or a trained DDIM. Taking a DDIM diffusion model containing 100 sampling steps as an example, during the reverse diffusion process, the sampling time steps, for example, range from the 99th sampling time step to the 0th sampling time step, to continuously denoise the noise, generate content, and output it. The first sampling time step may include any of the first N sampling time steps, and the second sampling time step may include any of the last M sampling time steps. Here, M and N are both positive integers, and M and N may be equal or unequal. The sum of M and N may be the total number of sampling time steps or may be less than the total number of sampling time steps. Optionally, the proportion of the first N sampling time steps in the total sampling time steps does not exceed 50%. Optionally, the proportion of the first N sampling time steps in the total sampling time steps is 30% to 40%.

[0069] According to some embodiments of this application, the first randomness parameter and the second randomness parameter can be parameters of the same type with different values; or, the first randomness parameter and the second randomness parameter can be parameters of different types; as long as the first randomness parameter and the second randomness parameter can introduce different randomness or diversity into the corresponding sampling time steps. Using parameters of the same type can simplify the control logic. As an example, in the forward diffusion process of DDPM or DDIM, Gaussian noise can be gradually added to the data over T time steps, as expressed by the following formula:

[0070]

[0071] Where q(x) t |x t-1 ) represents the probability density function, indicating the probability density function of the noisy data x at time step t-1. t-1 Under the premise that the data at time step t is x t The probability distribution; N represents a Gaussian distribution, β t Indicates noise scheduling, used to control the intensity of noise addition, x t x represents the noisy data generated at time step t. t-1 This represents the noisy data generated at time step t-1, where I represents the identity matrix, T represents the number of time steps, and t∈[1,T].

[0072] In the reverse diffusion process of DDPM, for example, a random reverse sampling process can be used, as expressed by the following formula:

[0073]

[0074] Where, x t x represents the data at sampling time step t. t-1 Data representing sampling time step t-1; α t =1-β t The coefficient used to control the noise level at the corresponding sampling time step t can control the degree of noise addition during the diffusion process. Optionally, α t It can be the weight corresponding to the sampling time step t, used to determine the degree of data retention; The noise standard deviation coefficient used for the corresponding sampling time step t; ∈ θ (x t ,t) represents the model's prediction noise, indicating the noise level from x t The noise component predicted in the model, where θ represents the model parameters; (Following a Gaussian distribution) introduces randomness or diversity.

[0075] In the reverse diffusion process relative to DDPM In this embodiment of the application, β can be used as a means to... t The settings are used to further improve the robustness of the diffusion model.

[0076] In the reverse diffusion process of DDIM, for example, a deterministic reverse process with the noise term set to zero can be used to achieve faster sampling with fewer sampling time steps:

[0077]

[0078] In some embodiments of this application, for DDIM, the following formula can be further introduced to interpolate between random sampling and deterministic sampling using the parameter η, in order to further improve the robustness of the diffusion model:

[0079]

[0080] In formula (5), Satisfying formula (6), σ t Satisfies formula (7):

[0081]

[0082] When η = 0, the sampling process can be understood as deterministic sampling, such as standard DDIM, while when η = 1, random DDPM sampling is restored.

[0083] The meaning of each of the above parameters can be found in the description of the above embodiments.

[0084] In some embodiments of this application, the above-mentioned randomness parameters include, for example, parameter η. By adjusting parameter η at one or more sampling time steps of the diffusion model, the influence of hardware noise can be resisted, the directionality of error accumulation can be reduced, and thus the robustness of the diffusion model can be improved. Based on the foregoing research, modulating parameter η to introduce higher randomness or diversity in relatively early sampling time steps helps to weaken the directionality of error accumulation in those early stages. Conversely, modulating parameter η to introduce lower randomness or diversity in later sampling time steps helps to reduce the number of subsequent sampling time steps, thereby further improving the robustness and operating efficiency of the diffusion model. In other embodiments of this application, the above-mentioned randomness parameters may also include other parameters related to randomness or diversity, such as parameter σ. t wait.

[0085] According to some embodiments of this application, the diffusion model may include multiple sampling time steps, the first N sampling time steps of the multiple sampling time steps include a first sampling time step, and the last M sampling time steps of the multiple sampling time steps include a second sampling time step, where M and N are both positive integers.

[0086] According to some embodiments of this application, the randomness parameter used in the first N sampling time steps falls within a first value range, while the randomness parameter used in the last M sampling time steps is 0, wherein the minimum value in the first value range is greater than 0. Since the randomness parameter values ​​in the first N sampling time steps are all greater than 0, and the randomness parameter values ​​in the last M sampling time steps are 0, the randomness or diversity introduced in the first N sampling time steps is greater than that introduced in the last M sampling time steps. This can reduce the cumulative impact of hardware noise on weight perturbation in the relatively early stages of the reverse diffusion process, improving the robustness of the diffusion model. Furthermore, this control method has simpler control logic and better diffusion model operating efficiency, achieving a good balance between diffusion model operating efficiency and robustness.

[0087] According to some embodiments of this application, the first value range is, for example, 0.4 to 1. The randomness parameter used in any of the first N sampling time steps can be any value between 0.4 and 1, and the randomness parameter used in any of the last M sampling time steps is 0. Optionally, the first value range can be 0.6 to 0.8. This can further improve the robustness of the diffusion model.

[0088] As an example, in this embodiment of the application, the time-related scheduling parameter η(t) is defined as a piecewise function, as shown in formula (8):

[0089]

[0090] Wherein, η define This represents the set value of η, which can be the optimal value of η determined through experience, experimentation, or testing, while t threshold This can represent the boundary that distinguishes between sensitive and insensitive time steps. This formula ensures that η(t) is not zero for sensitive time steps (relatively earlier stage time steps) and is zero for insensitive time steps, thereby reducing control complexity and accelerating the reverse diffusion process of the diffusion model, further improving the operating efficiency of the diffusion model.

[0091] The sampling process corresponding to the control method of this application is as follows: (9)

[0092]

[0093] In formula (9), σ t (η t ) contains η tThis control method eliminates the need to modify the trained diffusion model or underlying hardware architecture, reducing application adaptation difficulty. Especially for in-memory computing architectures, this method facilitates the deployment of the diffusion model on such architectures and allows for direct reduction of hardware noise during inference through parameter adjustment, thus improving inference reliability. Adjusting randomness or diversity during inference effectively acts as a noise-aware regularization mechanism, mitigating the cumulative impact of hardware non-ideals while maintaining the high-quality generation capability of the diffusion model.

[0094] As an example, Figure 8 A schematic diagram comparing the sampling methods based on traditional DDPM and DDIM with the control method of this application is shown. Figure 8 As shown, in traditional DDPM, η = 1, and in DDIM, η = 0. The control method provided in this application introduces an η value that can vary according to the sampling time step based on DDIM, and it can have different values ​​at different sampling time steps.

[0095] According to some embodiments of this application, the first N sampling time steps account for no more than 50% of the total sampling time steps in the diffusion model.

[0096] According to some embodiments of this application, the first N sampling time steps account for 30% to 40% of the total sampling time steps in the diffusion model.

[0097] By reasonably setting the proportion of the first N sampling time steps in multiple sampling time steps, it is beneficial to improve the robustness of the diffusion model by increasing the randomness or diversity of the sampling time steps in the relatively early stages, while also improving the robustness and running efficiency of the diffusion model by reducing the number of sampling time steps.

[0098] According to some embodiments of this application, the randomness parameters of the first N sampling time steps may be the same or different.

[0099] As an example, the randomness parameter for the first N sampling time steps is the same value, for example, the randomness parameter for the first N sampling time steps is 0.6, 0.7, or 0.8, etc. This helps to simplify the control of the diffusion model.

[0100] As an example, the randomness parameters of the first N sampling time steps can be partially the same, partially different, or completely different.

[0101] Taking the first N sampling time steps as sampling time steps 99 to 60 as an example, we will explain why the randomness parameters of the first N sampling time steps are the same. For example, the randomness parameter of sampling time steps 99 to 70 is 0.8, and the randomness parameter of sampling time steps 69 to 60 is 0.5. Or, for example, the randomness parameter of sampling time steps 99 to 70 is 0.9, and the randomness parameters of sampling time steps 69 to 60 are different values ​​of 0.7 to 0.8. That is, the randomness parameters corresponding to sampling time steps 69 to 60 are all different.

[0102] Taking the first N sampling time steps as sampling time steps 99 to 60 as an example, we will explain why the randomness parameters of the first N sampling time steps are all different. For example, the randomness parameters of sampling time steps 99 to 60 are different values ​​between 0.4 and 1, and the randomness parameters corresponding to each sampling time step are different.

[0103] The above are merely examples and are not intended to limit the value or range of the randomness parameter.

[0104] According to some embodiments of this application, the number of sampling time steps in the diffusion model is less than or equal to a first threshold, such as 50, 80, 100, 120, 150, or 200. The applicant has found that reducing the number of sampling time steps not only does not decrease the quality of the generated content, but on the contrary, it can increase the quality of the generated content; for example, the cumulative error generated by sampling time steps in earlier stages can be reduced, thereby further improving the robustness of the diffusion model. Therefore, by setting the number of sampling time steps in the diffusion model to be less than or equal to the first threshold, and using the above control method based on this initial sampling time step, the robustness of the diffusion model can be further improved.

[0105] According to some embodiments, the control methods of this application can be used to perform performance tests on diffusion models. For example, diffusion models pre-trained on datasets such as CIFAR-10, CelebA-HQ, Church, and Bedroom are used for testing. It can be seen that the above control methods can effectively improve the robustness of the diffusion model.

[0106] To verify the effectiveness of the control method of this application, the embodiments of this application first use standard DDIM sampling as a baseline, and design an inverse method based on time-step sensitive dynamic sparse sampling technology. This method adjusts the sampling step size by increasing the step size of sensitive time steps and decreasing the step size of non-sensitive time steps. The step size adjustment is expressed as the following formula (10):

[0107]

[0108] Where S(t) represents the sensitivity of the time step, and α is used to control the effect of sensitivity on the time step.

[0109] Figure 9 The diagram illustrates the performance of the diffusion models corresponding to the control method provided in this application and three other sampling methods on different noise environments and datasets. The vertical axis represents the FID score, and the horizontal axis represents the noise standard deviation. Examples of noise standard deviations are provided: 0.00, 0.02, 0.04, 0.06, 0.08, and 0.10. A standard deviation of 0.00 indicates no added noise. The different datasets include the CIFAR-10 dataset, the CelebA-HQ dataset, the Church dataset, and the Bedroom dataset. The control method provided in this application uses the DDIM model of the control method provided in the above embodiments (taking 100 sampling time steps as an example). The diffusion models corresponding to the other three sampling methods are baseline method 1, baseline method 2 and inverse method, respectively. Baseline method 1 uses a DDIM model containing 100 sampling time steps (DDIM-100 steps), baseline method 2 (DDIM-20 steps) uses a DDIM model containing 20 sampling time steps, and the inverse method uses a DDIM model containing 100 sampling time steps.

[0110] like Figure 9 As shown, a comparative analysis of the two baseline methods, DDIM-100 and DDIM-20, demonstrates that reducing the number of sampling time steps enhances robustness, especially under high noise conditions. For example, at a noise level of 0.08, DDIM-20 achieved an average FID improvement of 7.9 across four datasets compared to DDIM-100. DDIM-20 exhibits superior performance in low-noise environments compared to noise-free conditions, a phenomenon attributable to the inherent loss of image detail generated by 20 time steps, where noise perturbations anomalously enhance age details and sharpness. From the perspective of noise variation, the control method provided in this application achieves optimal performance at almost all noise levels, and its advantages become increasingly apparent as noise intensity increases. For instance, in the CIFAR-10 dataset, even with a noise level increasing to 0.1, the control method provided in this application maintains an FID score of 9.47, while the FID score of DDIM-100 deteriorates to 34.75. Furthermore, the effectiveness of the control method provided in this application remains consistent across different datasets.

[0111] This application conducted extensive evaluations on four different datasets containing various image styles and resolutions. Specifically, the CelebA-HQ dataset, Church dataset, and Bedroom dataset have a resolution of 256x256 pixels, while CIFAR-10 uses 32x32 pixels. At a noise level of 0.1, the control method provided in this application's embodiments achieved average FID improvements of 27.7, 18.3, and 7.4, respectively, compared to DDIM-100 steps, DDIM-20 steps, and the inverse method.

[0112] To accurately assess the robustness of the diffusion model under real hardware conditions, this application also provides a comprehensive noise model that captures the main non-ideals in analog in-memory computing architectures. Figure 10 A schematic diagram illustrating the non-idealities included in the integrated noise model is shown, such as... Figure 10 The three figures in the first row correspond to the noise in the analog-to-digital conversion (ADC) process. Figure 10 The three figures in the second row correspond to device nonlinearity. Figure 10 The three figures in the third row correspond to programming noise.

[0113] 1) Programming noise reflects the fundamental accuracy limitations in setting synaptic weights in analog storage devices. During the weight programming phase, the variability of the programming process leads to a deviation between the expected weights and the actual stored weights. This embodiment of the application models this as additive Gaussian noise applied to each weight parameter:

[0114]

[0115] Among them, w target Represents the ideal weights, This represents the programming noise variance, which is related to the magnitude of the target weights and is determined by the programming precision of the underlying memory technology.

[0116] 2) ADC noise: When computation occurs in the analog domain, analog-to-digital conversion (ADC) can be performed, resulting in a digital output. The ADC process may introduce quantization errors and / or thermal noise, thereby reducing computational accuracy. In this embodiment, ADC noise is modeled as additive white Gaussian noise.

[0117]

[0118] Where y analog This represents the simulation results. This represents the ADC noise variance, which depends on the ADC resolution, sampling rate, and thermal conditions. Different ADC channels have different noise characteristics and may exhibit spatial correlation, depending on the physical layout of the memory cell array.

[0119] 3) Device nonlinearity: Actual analog devices exhibit nonlinear behavior, differing from the ideal linear operation assumed in digital implementations. This nonlinearity can manifest in memory cell characteristics and peripheral circuitry. Embodiments of this application use a polynomial approximation to model this systematic error:

[0120]

[0121] Where α2 and α3 represent the second and third nonlinear coefficients, respectively. Capture the changes between devices in the nonlinear response.

[0122] This application also tested the performance of the diffusion models corresponding to the control method provided in this application and the other three sampling methods under the above three hardware noise environments.

[0123] As an example, Figure 11 The illustration shows a performance comparison of the diffusion models corresponding to the control method provided in this application and three other sampling methods under three different hardware noise environments. The three different hardware noise environments are as follows: Figure 10 The diagram shows ADC noise, programming noise, and nonlinear noise. The diffusion models corresponding to the other three sampling methods are baseline method 1, baseline method 2, and the inverse method, respectively. Baseline method 1 uses a DDIM model with 100 sampling time steps, baseline method 2 uses a DDIM model with 20 sampling time steps, and the inverse method uses a DDIM model with 100 sampling time steps.

[0124] like Figure 11 As shown in the left figure, in any of the three noise environments, compared to the diffusion models corresponding to the other three sampling methods, the control method of this application embodiment has universal applicability to various types of noise. The performance of the model was evaluated under the influence of three different noise types. Compared with the baseline method, the control method provided by this application embodiment shows a significant improvement in FID, with an FID improvement range of approximately 1.9 to 5.6.

[0125] like Figure 11 As shown in the right figure, compared to the diffusion models corresponding to the other three sampling methods, the control method provided in this application embodiment exhibits significant advantages under the combined effects of the three noise types. For example, the control method provided in this application embodiment achieves an FID score of 11.5, which is superior to DDIM's FID of 17.4.

[0126] Table 1

[0127]

[0128] Table 1 shows the FID of the three methods under noisy and noisy conditions at different sampling time steps (ranging from 10 to 1000). The data indicates that, under noisy conditions, the control method provided in this application embodiment outperforms the baseline and inverse methods at different sampling time step lengths. With increasing sampling time step count, the control method provided in this application embodiment exhibits significantly enhanced robustness. This improvement can be attributed to the control method's ability to mitigate the impact of accumulated errors, which increase with the number of sampling steps. For example, at 500 sampling time steps, the FID of DDIM under noisy conditions rises to 163.38, indicating that the model is completely invalid, while the control method provided in this application embodiment still maintains an FID of 21.27, signifying a high level of content generation quality.

[0129] Furthermore, to further verify the advantages of the control method provided in this application compared to other methods, the performance of diffusion models trained on different datasets, including the Bedroom dataset, the CelebA-HQ dataset, and the Church dataset, was visualized. For 100-step and 20-step DDIM models trained on each dataset, the image quality was compared between the 100-step DDIM in a noise-free environment, the 100-step DDIM in a noisy environment, the 20-step DDIM in a noisy environment, and the diffusion model corresponding to the control method provided in this application. The results showed that the images generated by the 100-step DDIM in a noisy environment and the 20-step DDIM in a noisy environment exhibited significant color distortion and loss of image detail under noisy conditions. In contrast, the diffusion model corresponding to the control method provided in this application preserved image details to the greatest extent, such as maintaining clear and detailed faces.

[0130] In some embodiments, the number of sampling time steps with higher randomness introduced in the early stages of reverse diffusion, as mentioned above, is associated with the value of N in the first N sampling time steps including the first sampling time step, and the value of the randomness parameter. As an example, the values ​​of N and the randomness parameter that satisfy the FID performance requirements can be determined based on the FID of the diffusion model under multiple different values ​​of N and multiple different values ​​of the randomness parameter. For example, as shown in Table 2 below:

[0131] Table 2

[0132]

[0133] Table 2 shows the FID scores of the control method provided in this application embodiment on CIFAR data for different values ​​of η and different ranges of relatively early sampling time steps with higher randomness. The performance of the control method provided in this application embodiment was evaluated in two aspects: the variation of η and the range of relatively early sampling time steps. As the range of relatively early sampling time steps increases, i.e., the number of steps increases, a significant improvement in performance was observed, indicating that random sampling effectively mitigates the accumulation of error over time steps. The control method provided in this application embodiment is better with increasing η values, indicating that higher levels of random sampling are more effective against noise, further verifying the role of random sampling in correcting accumulated errors over time steps. Better performance was achieved on the CIFAR10 dataset with relatively early sampling time steps ranging from 100-70 and η = 0.8 or from 100-60 and η = 0.8.

[0134] In some embodiments of this application, by introducing deterministic sampling (e.g., a second sampling time step with a parameter of 0) during the reverse diffusion process, the robustness of the diffusion model is further improved, the operating efficiency of the diffusion model is further improved, and the control logic is simplified.

[0135] In the above method embodiments, the order of the process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] This application also provides a control device, which can be found in the embodiments of this application. Figure 12 . Figure 12 A schematic diagram of a control device according to an exemplary embodiment of this application is shown. Figure 12 As shown, the control device 1200 includes: at least one processor 1202 for executing the control method provided in any of the above embodiments; and a memory 1201 for storing data used by the at least one processor, such as randomness parameters for each sampling time step.

[0137] This application does not limit the type of control device. For example, according to some embodiments, the control device may be located in a terminal device, or it may be located in a server. For example, the control device may be located in a single-machine single-card server, a single-machine multi-card server, a distributed cluster server, or a cloud server, etc. The control device may also include software stack devices, etc.

[0138] A processor is a circuit with signal processing capabilities. For example, a processor can be a circuit with instruction read and execute capabilities. In other possible embodiments, the processor can implement its functions through the logical relationships of hardware circuits, which can be fixed or reconfigurable. For example, the processor is a hardware circuit implemented as an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document to configure the hardware circuit can be understood as the process of the processor loading instructions to achieve its functions. This application does not limit the type of processor, but includes, for example, a central processing unit (CPU), a microcontroller unit (MCU), a graphics processing unit (GPU), or a digital signal processor (DSP). Alternatively, the processor can include hardware circuits designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), or a deep learning processing unit (DPU).

[0139] This application also provides a computer program product, which includes instructions that, when executed by a processor, cause any of the control methods described in the above embodiments to be executed.

[0140] This application also provides a computer-readable medium storing instructions that, when executed by a processor, cause any of the control methods described in the above embodiments to be executed.

[0141] This application also provides an electronic device, which can be found in [reference 1]. Figure 13 . Figure 13 A schematic diagram of an electronic device according to an exemplary embodiment of this application is shown. Figure 13As shown, the electronic device 1300 may include any of the control devices described above. The electronic device may be equipped with a diffusion model, and the control device may be used to control the diffusion model, process the data of the electronic device, and generate content. This application does not limit the type of generated content, which may include video, voice, text, images, or a combination of one or more of the above. In some embodiments, the electronic device 1300 may include a memory computing system 1310 for accelerating the operation of the diffusion model. The electronic device 1300 may also include an input / output device 1320 for receiving user input or outputting processing results. This application does not limit the input type and output type. For example, input may include voice input, text input, image input, or video input. Output may include text output, voice output, image output, or video output. The electronic device may also include a processor 1330, which may process the data provided to the memory computing system 1310 or process the output data of the memory computing system 1310. The output of the input / output device 1320 may be based on the output of the processor 1330 or the output of the control device.

[0142] This application does not limit the type of electronic device. For example, according to some embodiments, the electronic device may include wearable devices. Wearable devices include, but are not limited to: head-mounted devices (e.g., helmets or hats), devices worn on the ears (e.g., headphones), devices worn on the wrist (e.g., watches), and devices worn on other parts of the body (e.g., electronic necklaces, medical monitoring devices, or glasses). According to some embodiments, the electronic device may include portable terminals. For example, the electronic device may include, but is not limited to, mobile phones, general-purpose computing devices (e.g., laptops or tablets), personal digital assistants, etc. According to some embodiments, the electronic device may include other types of edge devices, such as personal computers, in-vehicle computers or in-vehicle computing platforms, or smart home electronic products. According to some embodiments, the electronic device may also include devices such as servers.

[0143] In the above embodiments, the descriptions of different embodiments each have their own emphasis. Parts not described in detail or recorded in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the different embodiments described above can be freely combined as needed. Moreover, as technology evolves, the elements described in this application can be replaced by equivalent elements appearing after this application.

Claims

1. A control method characterized by, A method for controlling running of a diffusion model, the method comprising: controlling a first sampling time step of executing the diffusion model, the first sampling time step adopting a first randomness parameter; controlling a second sampling time step of executing the diffusion model, the second sampling time step adopting a second randomness parameter; wherein the first randomness parameter introduces more randomness than the second randomness parameter at the first sampling time step and the second sampling time step, and the first sampling time step is earlier than the second sampling time step.

2. The method of claim 1, wherein, The diffusion model comprises a plurality of sampling time steps, the first N sampling time steps of the plurality of sampling time steps comprise the first sampling time step, and the last M sampling time steps of the plurality of sampling time steps comprise the second sampling time step, where M and N are positive integers.

3. The method of claim 2, wherein, The randomness parameters adopted by the first N sampling time steps belong to a first value range, and the randomness parameters adopted by the last M sampling time steps have a value of 0, and the minimum value in the first value range is greater than 0.

4. The method according to claim 2 or 3, characterized in that, The proportion of the first N sampling time steps in the plurality of sampling time steps is no more than 50%.

5. The method of claim 4, wherein, The proportion of the first N sampling time steps in the plurality of sampling time steps is 30%-40%.

6. The method according to any one of claims 3 to 5, characterized in that, The randomness parameters of the N sampling time steps are the same or different.

7. The method according to any one of claims 1 to 6, characterized in that, The value range of the first randomness parameter is 0.4-1.

8. The method of claim 1, wherein, The number of sampling time steps of the diffusion model is less than or equal to a first threshold value.

9. A control device characterized by comprising: The control device comprises at least one processing circuit and a storage circuit, the processing circuit is used to execute the control method as claimed in any one of claims 1-8, and the storage circuit is used to store data for use by the processing circuit.

10. An electronic device, comprising: The control device as claimed in claim 9.

Images