Image reasoning method and device, equipment and storage medium

Abstract

The invention discloses an image reasoning method and device, equipment and a storage medium, which are applied to a diffusion model, and the method comprises the steps: obtaining an initial image, determining a current time step, and generating initial noise according to the initial image and the current time step; sampling is carried out according to the current time step and the initial noise to generate sampling features, and the sampling features comprise partial sampling features and complete sampling features; and generating a final image according to the current time step and the sampling features. A target linear combination is determined through a preset knowledge base to generate initial noise, so that an early sampling step of a diffusion denoising process is skipped, partial reasoning sampling or complete reasoning sampling can be determined to be performed according to different current time steps and the initial noise, complete reasoning is avoided by introducing cache features in partial time steps, and the accuracy and the reliability of the method are improved. The effect of approximate complete reasoning can be achieved only through a small part of reasoning, reasoning time and calculation expenditure are saved, and therefore the purpose of diffusion model acceleration is achieved.

Description

Technical Field

[0001] The present invention relates to the field of image diffusion technology, and in particular to an image reasoning method, device, equipment and storage medium. Background Art

[0002] With the popularity of diffusion models, their excellent generation capabilities have been widely used in various fields. Diffusion models simulate the image generation process through a series of random diffusion steps. The core concept is to start with random noise and gradually refine it until it is similar to the sample of the target distribution. In the process of reverse reasoning, the diffusion model continuously denoises a random complete noise image and finally generates a brand new image.

[0003] At present, the main technologies for reducing inference costs focus on reducing the number of inference steps [3,4,5,6] and using distillation, quantization, pruning and other technologies to reduce the model inference cost at each step [7,8,9]. However, the method of obtaining lightweight compressed models through distillation and quantization often requires large-scale retraining of the model. Although pruning can reduce the cost of making new lightweight models, it still requires a lot of computing resources and time, which leads to high inference cost and low efficiency. Summary of the invention

[0004] The present invention provides an image reasoning method, device, equipment and storage medium to accelerate the efficiency of diffusion model reasoning by caching intermediate states and improve the image reasoning speed.

[0005] According to one aspect of the present invention, there is provided an image reasoning method applied to a diffusion model, the method comprising:

[0006] Obtain an initial image, determine the current time step, and generate initial noise based on the initial image and the current time step;

[0007] Sampling is performed according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features;

[0008] Generate a final image based on the current time step and sampled features.

[0009] Optionally, initial noise is generated according to the initial image and the current time step, including: when the current time step is 0, early noise is generated according to the initial image; several steps of inference sampling are performed based on the early noise and the current time step to generate historical sampling features, and when the current time step reaches the specified time step, the historical sampling features are matched through a preset knowledge base to generate initial noise.

[0010] Optionally, historical sampling features are matched through a preset knowledge base to generate initial noise, including: determining whether the preset knowledge base contains a specified number of features that are similar to the historical sampling features, and if so, determining each linear combination of the specified number of features, taking the linear combination with the highest similarity to the historical sampling features among the linear combinations as the target linear combination, and determining the target weight corresponding to the target linear combination, calling the historical intermediate state of the specified stage of the trajectory where the target linear combination is located, and combining the historical intermediate state with the target weight to generate initial noise; otherwise, generating initial noise by random initialization based on the initial image.

[0011] Optionally, sampling is performed according to the current time step and the initial noise to generate a sampling feature, including: determining a cache feature call situation according to the current time step, wherein the cache feature call situation includes calling the cache feature and not calling the cache feature; when the cache feature call situation is calling the cache feature, obtaining the cache feature, performing partial inference sampling according to the initial noise and the cache feature to generate a partial sampling feature; when the cache feature call situation is not calling the cache feature, performing complete sampling according to the initial noise to generate a complete sampling feature.

[0012] Optionally, after performing complete sampling based on the initial noise, the method further includes: determining intermediate features generated during the complete sampling process; and storing the intermediate features to generate cache features.

[0013] Optionally, determining the cache feature call status according to the current time step includes: obtaining a preset time step sequence, wherein the preset time step sequence stores time steps that need to be fully sampled; determining whether the current time step is in the preset time step sequence, and if so, determining the cache feature call status as not calling the cache feature; otherwise, determining the cache feature call status as calling the cache feature.

[0014] Optionally, generating a final image according to the current time step and sampling features includes: determining whether the current time step reaches a preset time step threshold, and if so, generating a final image according to the sampling features; otherwise, performing inference sampling for the next time step.

[0015] According to another aspect of the present invention, there is provided an image reasoning device, which is applied to a diffusion model, and comprises:

[0016] An initial noise generation module is used to obtain an initial image, determine a current time step, and generate initial noise according to the initial image and the current time step;

[0017] A sampling feature generation module, used for performing sampling according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features;

[0018] The final image generation module is used to generate the final image according to the current time step and the sampling features.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] at least one processor;

[0021] and a memory communicatively coupled to the at least one processor;

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute an image reasoning method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an image reasoning method described in any embodiment of the present invention when executed.

[0024] The technical solution of the embodiment of the present invention generates initial noise by determining the target linear combination through a preset knowledge base, thereby skipping the early sampling steps of the diffusion denoising process, and determining whether to perform partial inference sampling or complete inference sampling according to different current time steps and initial noises. By introducing cache features in partial time steps to avoid performing complete inference, only a small part of the inference is needed to achieve an effect of approximate complete inference, saving inference time and computing overhead, thereby achieving the purpose of accelerating the diffusion model.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flowchart of an image reasoning method provided according to Embodiment 1 of the present invention;

[0028] Figure 2 is a flowchart of another image reasoning method provided according to the second embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of the structure of an image reasoning device provided according to Embodiment 3 of the present invention;

[0030] Figure 4 It is a structural schematic diagram of an electronic device for implementing an image reasoning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 A flowchart of an image reasoning method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a diffusion model is used to perform reverse reasoning on an image. The method can be executed by an image reasoning device, which can be implemented in the form of hardware and / or software. The image reasoning device can be configured in a computer controller. Figure 1 As shown, the method includes:

[0035] S110, acquiring an initial image, determining a current time step, and generating initial noise according to the initial image and the current time step.

[0036] Among them, the diffusion model simulates the image generation process through a series of random diffusion steps. Its core concept is to start from random noise and gradually refine it until it is similar to the sample of the target distribution. The diffusion process of the diffusion model includes forward reasoning and reverse reasoning. This embodiment is specifically applied to the reverse reasoning process of the diffusion model. In the process of reverse reasoning, the diffusion model continuously denoises a random complete noise image, and finally generates a brand new image. The initial image is the starting point of the entire image reasoning process. It is an existing image data and can be any type of image, such as a photo, a painting, etc. In the diffusion model, the initial image provides basic information and features for subsequent reasoning. The time step represents the different stages in the process of the image evolving from the initial state to the final state. As the time step increases, the image is gradually de-noised, and different time steps correspond to different degrees of noise levels and changes in image features. The initial noise is the initial sampling point generated based on the initial image and the current time step. This embodiment mainly includes two parts, namely, the prediction intermediate state part that directly skips the early reasoning sampling step by retrieving the intermediate state with the highest similarity in the preset knowledge base, and the cache intermediate state part that uses storage to call continuous approximate intermediate states to avoid repeated calculations.

[0037] Optionally, initial noise is generated according to the initial image and the current time step, including: when the current time step is 0, early noise is generated according to the initial image; several steps of inference sampling are performed based on the early noise and the current time step to generate historical sampling features, and when the current time step reaches the specified time step, the historical sampling features are matched through a preset knowledge base to generate initial noise.

[0038] Among them, the time step represents the different stages of the image evolution from the initial state to the final state. If the current time step is 0, it means that it is in the initial state. When it is determined that the current time step is 0, an early noise can be obtained according to the initial image described by the prompt word. When the current time step reaches the specified time step, the controller can further obtain the historical sampling features, which refer to the sampling features of the previous few steps of the current time step, and then match the historical sampling features through the preset knowledge base to generate the corresponding initial noise.

[0039] Optionally, historical sampling features are matched through a preset knowledge base to generate initial noise, including: determining whether the preset knowledge base contains a specified number of features that are similar to the historical sampling features, and if so, determining each linear combination of the specified number of features, taking the linear combination with the highest similarity to the historical sampling features among the linear combinations as the target linear combination, and determining the target weight corresponding to the target linear combination, calling the historical intermediate state of the specified stage of the trajectory where the target linear combination is located, and combining the historical intermediate state with the target weight to generate initial noise; otherwise, generating initial noise by random initialization based on the initial image.

[0040] Specifically, the preset knowledge base is a pre-built collection that stores various feature information. The controller can search the database to query whether there is a specified number of features that are closest to the historical sampling features. If there are approximate features in the knowledge base, the controller will further determine the linear combinations of the specified number of features. A linear combination is a new feature obtained by weighted summation of multiple features. Various different linear combinations can be obtained through different weight combinations. Then the similarity between each linear combination and the historical sampling features is calculated, and the linear combination with the highest similarity is determined as the target linear combination. After determining the target linear combination, it is also necessary to determine the target weight corresponding to the combination. The weight determines the contribution of each feature in the combination. Among them, the controller refers to a computer controller that performs image reasoning.

[0041] In the diffusion model, the diffusion process is usually a step-by-step trajectory, and each stage has a corresponding intermediate state. The specified stage refers to the middle and late stages of the trajectory. The controller will call the historical intermediate states of the middle and late stages of the trajectory where the target linear combination is located, combine the historical intermediate states with the target weight, and use the combination as the initial point of the remaining steps of the sampling process, that is, the initial noise. By using the preset knowledge base for retrieval, the existing information can be used to generate the initial noise, thereby saving computational overhead and improving reasoning efficiency.

[0042] Specifically, if the preset knowledge base does not contain a specified number of features that are similar to the historical sampling features, then the information in the knowledge base cannot be used to generate the initial noise. In this case, the controller will randomly initialize the initial noise based on the initial image.

[0043] S120. Sampling is performed according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features.

[0044] In image reasoning, sampling is performed based on the current time step and initial noise to generate sampling features. Through sampling, a part of the features can be extracted from the noise distribution to generate subsequent images. Sampling features include partial sampling features and complete sampling features.

[0045] S130 , generating a final image according to the current time step and sampling features.

[0046] Optionally, generating a final image according to the current time step and sampling features includes: determining whether the current time step reaches a preset time step threshold, and if so, generating a final image according to the sampling features; otherwise, performing inference sampling for the next time step.

[0047] It should be noted that in the diffusion model, the advancement of the time step represents the evolution of the image from the initial noisy state to the final clear image state. The preset time step threshold is a pre-set time step value, which marks the stage when the image generation process is close to completion. By judging whether the current time step reaches the preset time step threshold, the progress of image generation can be determined.

[0048] Specifically, when the current time step reaches a preset time step threshold, the controller can generate a final image based on the sampled features. The sampled features are image features obtained through the sampling process in the previous time step. The process of generating the final image usually involves the integration and processing of the sampled features, for example, it can include merging partial sampled features with complete sampled features, weighted summing of features, mapping through a neural network, and other operations to obtain a final clear image.

[0049] It can be known that if the current time step does not reach the preset time step threshold, it means that the image generation process is not yet complete and it is necessary to continue the inference sampling of the next time step. The inference sampling process of the next time step is similar to that of the previous time step. According to the current image state (which may be a partially denoised image or a noisy image) and the current time step, sampling is performed to generate new sampling features. This process is repeated continuously. As the time step progresses, the image gradually becomes clearer and the sampling features become closer and closer to the features of the final image.

[0050] The technical solution of the embodiment of the present invention generates initial noise by determining the target linear combination through a preset knowledge base, thereby skipping the early sampling steps of the diffusion denoising process, and determining whether to perform partial inference sampling or complete inference sampling according to different current time steps and initial noises. By introducing cache features in partial time steps to avoid performing complete inference, only a small part of the inference is needed to achieve an effect of approximate complete inference, saving inference time and computing overhead, thereby achieving the purpose of accelerating the diffusion model.

[0051] Embodiment 2

[0052] Figure 2 This is a flowchart of an image inference method provided in the second embodiment of the present invention. This embodiment adds a specific process of sampling according to the current time step and the initial noise to generate sampling features based on the above-mentioned first embodiment. Among them, the specific contents of steps S210 and S250 are roughly the same as steps S110 and S130 in the first embodiment, so they will not be repeated in this embodiment. Figure 2 As shown, the method includes:

[0053] S210, acquiring an initial image, determining a current time step, and generating initial noise according to the initial image and the current time step.

[0054] Optionally, initial noise is generated according to the initial image and the current time step, including: when the current time step is 0, early noise is generated according to the initial image; several steps of inference sampling are performed based on the early noise and the current time step to generate historical sampling features, and when the current time step reaches the specified time step, the historical sampling features are matched through a preset knowledge base to generate initial noise.

[0055] Optionally, historical sampling features are matched through a preset knowledge base to generate initial noise, including: determining whether the preset knowledge base contains a specified number of features that are similar to the historical sampling features, and if so, determining each linear combination of the specified number of features, taking the linear combination with the highest similarity to the historical sampling features among the linear combinations as the target linear combination, and determining the target weight corresponding to the target linear combination, calling the historical intermediate state of the specified stage of the trajectory where the target linear combination is located, and combining the historical intermediate state with the target weight to generate initial noise; otherwise, generating initial noise by random initialization based on the initial image.

[0056] S220. Determine a cache feature calling situation according to the current time step, wherein the cache feature calling situation includes calling the cache feature and not calling the cache feature.

[0057] Optionally, determining the cache feature call status according to the current time step includes: obtaining a preset time step sequence, wherein the preset time step sequence stores time steps that need to be fully sampled; determining whether the current time step is in the preset time step sequence, and if so, determining the cache feature call status as not calling the cache feature; otherwise, determining the cache feature call status as calling the cache feature.

[0058] The preset time step sequence is a set of pre-set time steps, which stores the time steps that need to be fully sampled. For example, in the diffusion model, different time steps correspond to different stages of the image evolution from the initial state to the final state. Some time steps may need to be fully sampled to obtain more randomness and diversity, or to better capture certain specific features of the image. By storing these time steps in the preset time step sequence, it is convenient to determine whether full sampling is required in the subsequent process.

[0059] Specifically, if the current time step is in the preset time step sequence, then the cache feature call situation is determined as not calling the cache feature. At this time, the controller needs to perform complete sampling and does not rely on the previous cache feature. If the current time step is not in the preset time step sequence, then the cache feature call situation is determined as calling the cache feature. At this time, the previously calculated and stored cache features can be used, avoiding repeated calculations, saving computing resources and time, and thus improving the efficiency of sampling.

[0060] In summary, this implementation can flexibly choose whether to call cache features according to different time step requirements to improve image reasoning efficiency.

[0061] S230: When the cache feature calling situation is calling the cache feature, the cache feature is obtained, and partial inference sampling is performed according to the initial noise and the cache feature to generate partial sampling features.

[0062] Among them, when the cache feature call situation is to call the cache feature, the controller will first obtain the cache feature, which is calculated and stored in the previous time step. Then the controller will combine the initial noise and the cache feature for partial inference sampling. Partial inference sampling means sampling only part of the image instead of fully sampling the entire image. Since the cache feature has already provided part of the information, only the undetermined part needs to be sampled.

[0063] S240: When the cache feature calling situation is that the cache feature is not called, complete sampling is performed according to the initial noise to generate a complete sampling feature.

[0064] Specifically, if it is determined not to call the cache feature, the controller will perform a complete sampling based on the initial noise. Complete sampling means that the entire image is fully sampled without relying on cache information.

[0065] Optionally, after performing complete sampling based on the initial noise, the method further includes: determining intermediate features generated during the complete sampling process; and storing the intermediate features to generate cache features.

[0066] In the process of complete sampling based on the initial noise, the controller will use the high-level features as intermediate features. The purpose of storing the determined intermediate features is to call the cache features in the subsequent sampling process to improve the sampling efficiency. By storing the intermediate features, it is possible to avoid recalculating the same information every time sampling is performed, thereby saving computing resources and time.

[0067] Specifically, the generated cache features can be stored in the form of various data structures. In the subsequent sampling process, when the cache features need to be called, the corresponding features can be quickly obtained according to specific indexes or query conditions.

[0068] S250, generating a final image according to the current time step and sampling features.

[0069] Optionally, generating a final image according to the current time step and sampling features includes: determining whether the current time step reaches a preset time step threshold, and if so, generating a final image according to the sampling features; otherwise, performing inference sampling for the next time step.

[0070] The technical solution of the embodiment of the present invention generates initial noise by determining the target linear combination through a preset knowledge base, thereby skipping the early sampling steps of the diffusion denoising process, and determining whether to perform partial inference sampling or complete inference sampling according to different current time steps and initial noises. By introducing cache features in partial time steps to avoid performing complete inference, only a small part of the inference is needed to achieve an effect of approximate complete inference, saving inference time and computing overhead, thereby achieving the purpose of accelerating the diffusion model.

[0071] Embodiment 3

[0072] Figure 3 This is a schematic diagram of the structure of an image reasoning device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an initial noise generation module 310, which is used to obtain an initial image, determine a current time step, and generate initial noise according to the initial image and the current time step;

[0073] A sampling feature generation module 320, used for performing sampling according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features;

[0074] The final image generation module 330 is used to generate a final image according to the current time step and the sampling features.

[0075] Optionally, the initial noise generation module 310 specifically includes: an initial noise generation unit, used to: when the current time step is 0, generate early noise according to the initial image; perform several steps of inference sampling based on the early noise and the current time step to generate historical sampling features, and when the current time step reaches the specified time step, match the historical sampling features through a preset knowledge base to generate initial noise.

[0076] Optionally, the initial noise generation unit specifically includes: a knowledge base matching subunit, used to: determine whether the preset knowledge base contains a specified number of features that are similar to the historical sampling features, and if so, determine each linear combination of the specified number of features, take the linear combination with the highest similarity to the historical sampling features among the linear combinations as the target linear combination, and determine the target weight corresponding to the target linear combination, call the historical intermediate state of the specified stage of the trajectory where the target linear combination is located, and combine the historical intermediate state with the target weight to generate initial noise; otherwise, generate initial noise by random initialization based on the initial image.

[0077] Optionally, the sampling feature generation module 320 is specifically used to: determine the cache feature calling situation according to the current time step, wherein the cache feature calling situation includes calling the cache feature and not calling the cache feature; when the cache feature calling situation is calling the cache feature, obtain the cache feature, perform partial inference sampling based on the initial noise and the cache feature to generate a partial sampling feature; when the cache feature calling situation is not calling the cache feature, perform complete sampling based on the initial noise to generate a complete sampling feature.

[0078] Optionally, the device also includes: a cache feature generation module, which is used to determine the intermediate features generated in the complete sampling process after complete sampling according to the initial noise; and store the intermediate features to generate cache features.

[0079] Optionally, the sampling feature generation module 320 specifically includes: a cache feature call status determination unit, which is used to: obtain a preset time step sequence, wherein the preset time step sequence stores time steps that need to be fully sampled; determine whether the current time step is in the preset time step sequence, and if so, determine that the cache feature call status is not calling the cache feature; otherwise, determine that the cache feature call status is calling the cache feature.

[0080] Optionally, the final image generation module 330 is specifically used to: determine whether the current time step reaches a preset time step threshold, and if so, generate a final image according to the sampling characteristics; otherwise, perform inference sampling for the next time step.

[0081] The technical solution of the embodiment of the present invention generates initial noise by determining the target linear combination through a preset knowledge base, thereby skipping the early sampling steps of the diffusion denoising process, and determining whether to perform partial inference sampling or complete inference sampling according to different current time steps and initial noises. By introducing cache features in partial time steps to avoid performing complete inference, only a small part of the inference is needed to achieve an effect of approximate complete inference, saving inference time and computing overhead, thereby achieving the purpose of accelerating the diffusion model.

[0082] An image reasoning device provided by an embodiment of the present invention can execute an image reasoning method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0083] Embodiment 4

[0084] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0085] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image reasoning method.

[0088] In some embodiments, an image reasoning method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image reasoning method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform an image reasoning method in any other appropriate manner (e.g., by means of firmware).

[0089] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0091] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0092] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0093] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0094] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0095] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image reasoning method, applied to a diffusion model, characterized in that: include: Acquire an initial image, determine a current time step, and generate initial noise according to the initial image and the current time step; Sampling is performed according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features; A final image is generated according to the current time step and the sampled features.

2. The method according to claim 1, characterized in that The generating initial noise according to the initial image and the current time step comprises: When the current time step is 0, generating early noise according to the initial image; Several steps of inference sampling are performed based on the early noise and the current time step to generate historical sampling features. When the current time step reaches a specified time step, the historical sampling features are matched through a preset knowledge base to generate initial noise.

3. The method according to claim 2, characterized in that The matching of the historical sampling features by a preset knowledge base to generate initial noise includes: Determine whether the preset knowledge base contains a specified number of features that are similar to the historical sampling features. If so, determine each linear combination of the specified number of features, use the linear combination with the highest similarity to the historical sampling features among the linear combinations as the target linear combination, determine the target weight corresponding to the target linear combination, call the historical intermediate state of the specified stage of the trajectory where the target linear combination is located, and combine the historical intermediate state with the target weight to generate initial noise; Otherwise, the initial noise is generated by random initialization according to the initial image.

4. The method according to claim 1, characterized in that: The sampling is performed according to the current time step and the initial noise to generate a sampling feature, comprising: Determine a cache feature calling condition according to the current time step, wherein the cache feature calling condition includes calling the cache feature and not calling the cache feature; When the cache feature calling situation is calling the cache feature, obtaining the cache feature, performing partial inference sampling according to the initial noise and the cache feature to generate partial sampling features; When the cache feature calling situation is not calling the cache feature, complete sampling is performed according to the initial noise to generate a complete sampling feature.

5. The method according to claim 4, characterized in that After the complete sampling is performed according to the initial noise, the method further includes: Identify intermediate features produced during the complete sampling process; The intermediate features are stored to generate the cache features.

6. The method according to claim 4, characterized in that The determining of the cache feature calling condition according to the current time step includes: Obtaining a preset time step sequence, wherein the preset time step sequence stores time steps that need to be fully sampled; Determine whether the current time step is in the preset time step sequence, and if so, determine that the cache feature calling condition is not to call the cache feature; Otherwise, it is determined that the cache feature calling condition is calling the cache feature.

7. The method according to claim 1, characterized in that The generating a final image according to the current time step and the sampling features comprises: Determine whether the current time step reaches a preset time step threshold, and if so, generate a final image according to the sampling features; Otherwise, perform inference sampling for the next time step.

8. An image reasoning device, applied to a diffusion model, characterized in that: include: An initial noise generation module, used to obtain an initial image, determine a current time step, and generate initial noise according to the initial image and the current time step; A sampling feature generation module, used for performing sampling according to the current time step and the initial noise to generate sampling features, wherein the sampling features include partial sampling features and complete sampling features; The final image generation module is used to generate a final image according to the current time step and the sampling features.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images