Image processing apparatus and method, model training apparatus, and storage medium

Abstract

The invention discloses an image processing apparatus and method, a model training apparatus, and a computer readable storage medium. The image processing apparatus may include an image generation unit configured to set a plurality of motion-related parameters to input data to vary the input data, and generate a second image based on the input data during the variation using a pre-trained diffusion generation model. The plurality of motion-related parameters include displacement, velocity, and acceleration. The pre-trained diffusion generation model is obtained by the following steps: setting a plurality of motion related parameters for each pixel point of each original training image in a training image set to change the pixel points, and training a score-based diffusion generation model based on the changed training images of the plurality of stages in the change process to obtain a pre-trained diffusion generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing device and method, a model training device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of science and technology, image-based technologies (such as image recognition) are widely used. Summary of the invention

[0003] A brief overview of the disclosure is given below in order to provide a basic understanding of certain aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give certain concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0004] The purpose of the present disclosure is to provide an improved image processing device and method, a model training device, and a computer-readable storage medium.

[0005] According to one aspect of the present disclosure, an image processing device is provided, comprising: an image generation unit, configured to set a plurality of motion-related parameters for input data to change the input data, and to generate a second image based on the input data in the change process using a pre-trained diffusion generation model. The plurality of motion-related parameters include displacement, velocity, and acceleration. The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel point of each original training image in a training image set to change the pixel point, and training a score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model.

[0006] According to another aspect of the present disclosure, a model training device is provided, comprising a processing unit configured to: set a plurality of motion-related parameters for each pixel of each original training image in a training image set to change the pixel, and train a score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain a trained diffusion generation model for generating images. The plurality of motion-related parameters include displacement, velocity, and acceleration.

[0007] According to another aspect of the present disclosure, there is provided an image processing method, comprising: setting a plurality of motion-related parameters for input data to change the input data, and generating a second image based on the input data in the change process using a pre-trained diffusion generation model. The plurality of motion-related parameters include displacement, velocity, and acceleration. The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel point of each original training image in a training image set to change the pixel point, and training a score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model.

[0008] According to other aspects of the present disclosure, a computer program code and a computer program product for implementing the method according to the present disclosure are also provided, as well as a computer-readable storage medium having the computer program code for implementing the method according to the present disclosure recorded thereon.

[0009] Other aspects of the embodiments of the present disclosure are given in the following description, wherein the detailed description is used to fully disclose the preferred embodiments of the embodiments of the present disclosure without imposing limitations thereon. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure may be better understood by referring to the detailed description given below in conjunction with the accompanying drawings, wherein the same or similar reference numerals are used throughout the drawings to represent the same or similar components. The accompanying drawings, together with the following detailed description, are included in and form a part of the present specification to further illustrate the preferred embodiments of the present disclosure and to explain the principles and advantages of the present disclosure. Among them:

[0011] Figure 1 is a block diagram showing a functional configuration example of a model training device according to an embodiment of the present disclosure;

[0012] Figure 2 is a block diagram showing a functional configuration example of an image processing apparatus according to an embodiment of the present disclosure;

[0013] Figure 3A shows the distribution of some stages when a plurality of 2D Swiss rolls are generated using the image generating apparatus 200 according to an embodiment of the present disclosure;

[0014] Figure 3BThe distribution of partial phases in the case of generating multiple 2D Swiss rolls using a technique based on critically-damped Langevin diffusion (CLD) (see, for example, Reference 1: T. Dockhorn, A. Vahdat, and K. Kreis. Score-based generative modeling with critically-damped Langevindiffusion);

[0015] Figure 4A , Figure 4B and Figure 4C Examples of 2D Swiss rolls generated by using an image generating apparatus according to an embodiment of the present disclosure, a technique based on VP-SDE (Variance Preserving SDE) (see, for example, Reference 2: Y. Song, J. Sohl-Dickstein, DP Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations), and a technique based on CLD are respectively shown;

[0016] Figure 5 is a diagram showing a comparison between an image generating apparatus according to an embodiment of the present disclosure, a technique based on VP-SDE, and a technique based on CLD;

[0017] Figure 6 is a block diagram showing a functional configuration example of an image processing apparatus according to an embodiment of the present disclosure;

[0018] Figure 7 is a flowchart showing an example of a process of a model training method according to an embodiment of the present disclosure;

[0019] Figure 8 is a flowchart showing an example of the flow of an image processing method according to an embodiment of the present disclosure;

[0020] Fig. 9 is a flowchart showing an example of the flow of an image processing method according to an embodiment of the present disclosure; and

[0021] Fig.10 : is a block diagram showing an example structure of a personal computer that can be employed in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.

[0023] It is also necessary to explain here that in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not closely related to the present disclosure are omitted.

[0024] Embodiments according to various aspects of the present disclosure are described in detail below with reference to the accompanying drawings.

[0025] Embodiments of the first aspect

[0026] Figure 1 1 is a block diagram showing a functional configuration example of a model training device 100 according to an embodiment of the present disclosure. Figure 1 As shown, the model training device 100 according to an embodiment of the present disclosure may include a processing unit 102.

[0027] The processing unit 102 may set a plurality of motion-related parameters for each pixel of each original training image in the training image set to change the pixel, and train a score-based diffusion generation model (see, for example, reference 2) based on the changed training images at multiple stages in the change process (also referred to as the "diffusion process") to obtain a trained score-based diffusion generation model (also referred to as the "trained diffusion generation model"). The trained diffusion generation model may be used for image generation.

[0028] The above-mentioned multiple motion-related parameters include displacement, velocity and acceleration. Compared with the technology of only setting displacement and / or velocity, the obtained trained diffusion generation model has better performance, such as higher quality and better diversity of generated images.

[0029] The processing unit 102 may set noise to one or more of the above-mentioned multiple motion-related parameters as needed during the change process. In some examples, the processing unit 102 may apply random noise to only the acceleration among the multiple motion-related parameters, thereby further improving the smoothness and speed of the change process.

[0030] For example, the process of changing the pixel point can be modeled by the third-order Langevin dynamic equation. For example, when random noise is applied to only the acceleration among multiple motion-related parameters, the process of changing the pixel point can be expressed by the following equation (1):

[0031]

[0032] In formula (1), denote the position, velocity and acceleration of each pixel at time t, respectively; L is the Lipschitz constant of U; γ>0 denotes the friction coefficient; ξ>0 denotes the algorithmic parameter; U(q t ): is the potential function associated with the position vector, represents Brownian motion, associated with random noise.

[0033] From equation (1), we can see intuitively that random noise can only directly affect the acceleration s t , and the acceleration s t Through speed p t Indirectly affects the position q t , so q t The change of is relatively smooth. t In contrast, in the case of the process represented by equation (1), the position q t The smoothness of the change trajectory is two orders of magnitude higher.

[0034] As can be understood by those skilled in the art, by comparing the equation (1) with dq t 、dp t and / or ds t The corresponding expressions add or remove noise components, which can change the motion-related parameters to which the noise is applied.

[0035] For example, we can set U(q t )=L / 2||q t ||. For example, the changed image at time t can be expressed as Therefore, the above formula (1) can be written in matrix form dx t =f(t)x t dt+G(t)dw t , where f(t) and G(t) can be expressed by the following equations (2) and (3), respectively:

[0036]

[0037]

[0038] In the above formulas (2) and (3), represents the Kronecker product, I d In order to reduce the computational power, especially the complexity and expressiveness, the inventors conducted multiple experiments and used In this case, f(t) has three different real eigenvalues ​​in an elegant and simple closed form, namely, and So, x t The mean and variance of the transition density of become easy to compute. For example, x t The mean μ of the transition density t and variance∑ t The relationship can be expressed by the following equations (4) and (5):

[0039]

[0040]

[0041] In formula (4), p(x t |x0,t) represents the distribution of state x at time t, and N represents the normal distribution. In formula (5), represents the variance of i and j at time t. It can be seen that the prior distribution (or equilibrium distribution) of the diffusion process is Indicates that the final position, velocity, and acceleration are all Gaussian distributed.

[0042] When equation (1) is expressed by equation (2) and equation (3), there are two Hamiltonian components: and and an Ornstein-Uhlenbeck (OU) process This will make the diffusion process corresponding to equation (1) proceed quickly.

[0043] The key to the training process is to obtain the score function. Based on formula (4), the score function It can be expressed by the following formula (6):

[0044]

[0045] In formula (6), the noise is the covariance matrix∑ tCholesky factorization of (see, e.g., https: / / en.wikipedia.org / wiki / Cholesky_decomposition).

[0046] Because the noise ∈ 3d Always confined to a small bounding box, predicting noise can further improve the performance of the obtained trained model relative to the predicted score itself. The case of predicting noise during training can be equivalent to the case of training a score-based diffusion generative model based on the difference between the transformed training images and the original training images at multiple stages.

[0047] For example, we can assume that μ0=(q0,0 d , 0 d ), Among them, α<<1. So, p(x t |x0,t)=p(x t |q0, t). In this case, the score function based on equation (6) It can be expressed by the following formula (7):

[0048]

[0049] Therefore, the score function can be obtained by optimizing the denoising score matching loss (DSM loss) expressed by the following equation (8):

[0050]

[0051] In formula (8), In addition, the scale adjustment factor l t Satisfies the following formula (9):

[0052]

[0053] For example, the initial distribution b0 can be denoised for only a portion of all blocks of x0 (i.e., q0, p0, s0) and the distribution of the remaining blocks can be marginalized, so that the loss expressed by the following equation (10) can be obtained:

[0054]

[0055] For the above equation (10), the only requirement is that b0 must contain the block of interest q0. The above equation (10) can be further written as the following equation (11):

[0056]

[0057] By optimizing the loss expressed in equation (11), the score function can be obtained. Compared with the loss expressed in equation (8), the optimization process is easier.

[0058] In some embodiments, in addition to displacement, velocity and acceleration, other motion-related parameters may be included, thereby further improving the performance of the obtained trained diffusion model.

[0059] For example, in this case, the process of changing the pixel point can be expressed by the following equation (12):

[0060]

[0061] In equation (12), z1, z2 and z3 may correspond to position, velocity and acceleration, respectively.

[0062] Embodiments of the second aspect

[0063] Figure 2 2 is a block diagram showing a functional configuration example of the image processing apparatus 200 according to an embodiment of the present disclosure. Figure 2 As shown, the image processing apparatus 200 according to an embodiment of the present disclosure may include an image generating unit 202 .

[0064] The image generation unit 202 may be configured to set a plurality of motion-related parameters for the input data to change the input data, and generate a second image based on the input data in the change process using a pre-trained diffusion generation model. The plurality of motion-related parameters may include displacement, velocity, and acceleration. For example, the input data may be noise, a specific data distribution, etc.

[0065] For example, the pre-trained diffusion generation model is obtained by setting the above-mentioned multiple motion-related parameters for each pixel point of each original training image in the training image set to change the pixel point, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model. For example, the pre-trained diffusion generation model can be a trained diffusion generation model obtained by training through the model training device 100 described above.

[0066] By setting a plurality of motion-related parameters including displacement, velocity and acceleration, compared with only setting displacement and / or velocity, for example, the obtained second image has higher quality, better diversity, etc.

[0067] The image generation unit 202 may set noise to one or more of the above-mentioned multiple motion-related parameters as needed during the change process. In some examples, the image generation unit 202 may apply random noise to only the acceleration among the multiple motion-related parameters, thereby further improving the smoothness and speed of the change process, thereby further improving the quality of the second image, etc.

[0068] For example, the operation of setting multiple motion-related parameters for input data to change the input data can be expressed by the following formula (12):

[0069] dq t =-p t dt,

[0070] dp t =q t dt-γs t dt,

[0071]

[0072] In formula (13), represents the standard Wiener process, Indicates fractions The optimized predictor (also called “score function”) aims to denoise the prior distribution obtained through the training process into the target data.

[0073] Assume p(x0,0) is the initial distribution (i.e., the input data with initial velocity and acceleration), then the distribution p(x0,0) at time t during the change process is t , t) (i.e., the data in the changing process) can be expressed as in, is the Fokker-Planck operator.

[0074] The above formula (13) can be written as the following formula (14):

[0075]

[0076] It can be seen that equation (14) is very similar to equation (1), so many calculations about equation (1) can be extended to equation (14), and the remaining vector field can be expressed by the following equation (15):

[0077]

[0078] set up and are the Fokker-Planck operators corresponding to equations (14) and (15), respectively. So we can get and Based on Trotter theory (see, for example, reference 3: HFTrotter. On the product of semi-groups of operators. Proceedings of the American Mathematical Society), and Satisfies the following formula (16):

[0079]

[0080] In formula (16), P is a positive integer.

[0081] In solving In the process, it is necessary to calculate the mean and variance of the data after t / 2P time. Is through The mean value u is obtained by integrating the following equations (17) and (18): t and variance∑ t Gaussian distribution of:

[0082]

[0083]

[0084] Wherein, l(t) and M(t) are expressed by the following equations (19) and (20), respectively:

[0085]

[0086]

[0087] For example, the following formula (21) based on the one-step Euler method (see, for example, reference 2) can be used to calculate

[0088]

[0089] Figure 3A Distribution of some stages in a case where a plurality of 2D Swiss rolls are generated using the image generating apparatus 200 according to an embodiment of the present disclosure is shown. Figure 3B The distribution of some stages when multiple 2D Swiss rolls are generated using CLD-based technology. Figure 3A and Figure 3BIt can be seen that the image generation device 200 according to the embodiment of the present disclosure has a better synthesis effect and a more stable generation speed than the CLD-based technology. For example, for the image generation device 200 according to the embodiment of the present disclosure, in step 699, the sampling points have been gathered around five cluster centers, while for the CLD-based technology, in step 699, the sampling points still look like noise.

[0090] Figure 4A , Figure 4B and Figure 4C Examples of 2D Swiss rolls generated by using the image generating apparatus 200 according to an embodiment of the present disclosure, the VP-SDE (Variance Preserving SDE)-based technology, and the CLD-based technology are respectively shown. Figure 4A , Figure 4B and Figure 4C It can be seen that, compared with the VP-SDE-based technology and the CLD-based technology, the Swiss roll generated by the image generating device 200 according to the embodiment of the present disclosure has higher quality.

[0091] Figure 5 2 is a diagram showing a comparison between the image generating apparatus 200 according to an embodiment of the present disclosure, the VP-SDE-based technique, and the CLD-based technique. Figure 5 In , n represents the function evaluation. Figure 5 It can be seen that the FID (Fréchet Inception Distance) corresponding to the image generating device 200 is lower than both the VP-SDE-based technology and the CLD-based technology. This shows that compared with the VP-SDE-based technology and the CLD-based technology, the image generated by the image generating device 200 is closer to the real image.

[0092] In addition, the inventors have found through experiments that the image generation device 200 according to the embodiment of the present disclosure is not very sensitive to the hyperparameters L and α (initial velocity and acceleration variable scales), and can therefore be easily transplanted to other generation tasks, such as 3D data generation tasks.

[0093] In some examples, the image generation unit 202 can generate the second image based on the input data in the changing process under the guidance of the first image. The first image can be set according to the needs of the user. As an example, in the case where the image generation unit 202 is used to generate an image of a certain type of object (e.g., an object, an animal, etc.), the first image can be an image corresponding to the type of object.

[0094] As another example, the image generation unit 202 may be configured to generate a second version of an image (an example of a second image) based on a first version of an image (an example of a first image). For example, the first version of an image may be an image with a low resolution, and the second version of an image may be an image with a high resolution. For example, the first version of an image may be a blurred or occluded image, and the second version of an image may be an image with a clearer or less occlusion (or even no occlusion).

[0095] For example, when the image generation unit 202 generates the second image under the guidance of the first image, the above equation (13) can be rewritten as the following equations (22)-(24):

[0096]

[0097]

[0098]

[0099] The above equation (23) is made under the assumption that p0, s0 = 0. y represents the first image, represents the second image, and σ represents the variance of the white noise added during the image degradation process.

[0100] Note that although some examples of formulas used in the present application are described above, in actual applications, those skilled in the art may adopt other formulas based on the principles disclosed in the present application.

[0101] Embodiments of the third aspect

[0102] Figure 6 6 is a block diagram showing a functional configuration example of an image processing device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the image processing apparatus 600 according to an embodiment of the present disclosure may include an image generating unit 602 and an image recognizing unit 604 .

[0103] The image generation unit 602 may be configured to set a plurality of motion-related parameters for the input data to change the input data, and generate a second image based on the input data in the change process using a pre-trained diffusion generation model. The plurality of motion-related parameters may include displacement, velocity, and acceleration. For example, the image generation unit 602 may have a function and configuration similar to that of the image generation unit 202, and thus will only be briefly described below.

[0104] The image recognition unit 604 may be configured to recognize an object (eg, a face, an object, an animal, etc.) involved in the second image generated by the image generation unit 602 .

[0105] For example, the image processing device 600 can be implemented as an electronic device (such as a mobile phone, an automatic cash register, etc.) or can be integrated into an electronic device. For example, the image generation unit 602 can generate a second image under the guidance of a first image acquired by an electronic device (such as an image acquisition device integrated in the electronic device). For example, when the first image acquired by the electronic device is blurred or blocked, the image generation unit 602 can generate a clear or less blocked (or even unblocked) second image corresponding to the first image, and provide the generated second image to the image recognition unit 604, thereby improving the image recognition accuracy.

[0106] For example, the image processing device 600 can be implemented as a cash register or integrated into a cash register. For example, the cash register can also include an image acquisition device for acquiring an image of an object. The image generation unit 602 can process the image of the object acquired by the image acquisition device to generate a second image, and the image recognition unit 604 can recognize the second image to obtain the category of the object for the cash register to determine the price of the object. In this way, even if the object is blocked or the image of the object is not clear, the category of the object can be accurately identified.

[0107] For example, the image processing device 600 can be implemented as an identity recognition device or integrated into an identity recognition device. For example, the cash register may also include an image acquisition device for acquiring an image of a person or a part thereof (e.g., a face). The image generation unit 602 may process the image acquired by the image acquisition device to generate a second image, and the image recognition unit 604 may recognize the second image to obtain the identity of the person. In this way, even if the acquired image is not clear, the identity of the person can be accurately recognized.

[0108] Embodiments of the fourth aspect

[0109] Figure 7 is a flow chart showing an example of a process of a model training method according to an embodiment of the present disclosure. Figure 7 As shown, the model training method 700 according to an embodiment of the present disclosure may start at step S702 and end at step S706.

[0110] like Figure 7As shown, the model training method 700 may include a processing step S704. In the processing step S704, a plurality of motion-related parameters may be set for each pixel of each original training image in the training image set to change the pixel, and the score-based diffusion generation model may be trained based on the changed training images at multiple stages in the change process to obtain a trained score-based diffusion generation model. The trained diffusion generation model may be used for image generation. For example, the processing step S704 may be performed by the above-mentioned processing unit 102, and therefore will only be briefly described below.

[0111] The above-mentioned multiple motion-related parameters include displacement, velocity and acceleration. Compared with the technology of only setting displacement and / or velocity, the obtained trained diffusion generation model has better performance, such as higher quality and better diversity of generated images.

[0112] In some examples, random noise may be applied to only the acceleration among the multiple motion-related parameters, thereby improving the smoothness and speed of the change process.

[0113] Embodiments of the fifth aspect

[0114] Figure 8 FIG. 1 is a flowchart showing an example of the process of an image processing method according to an embodiment of the present disclosure. Figure 8 As shown, the image processing method 800 according to an embodiment of the present disclosure may start at step S802 and end at step S806.

[0115] like Figure 8 As shown, the image processing device 800 may include an image generating step S804. In the image generating step S804, a plurality of motion-related parameters may be set for the input data to change the input data, and a pre-trained diffusion generation model may be used to generate a second image based on the input data in the changing process. The plurality of motion-related parameters may include displacement, velocity, and acceleration. For example, the image generating step S804 may be performed by the image generating unit 202, so only a brief description is given below.

[0116] For example, the pre-trained diffusion generation model is obtained by setting the above-mentioned multiple motion-related parameters for each pixel point of each original training image in the training image set to change the pixel point, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model. For example, the pre-trained diffusion generation model can be a trained diffusion generation model obtained by training through the model training device 100 described above.

[0117] By setting a plurality of motion-related parameters including displacement, velocity and acceleration, compared with only setting displacement and / or velocity, for example, the obtained second image has higher quality, better diversity, etc.

[0118] Embodiments of the sixth aspect

[0119] Fig. 9 FIG. 1 is a flowchart showing an example of the process of an image processing method according to an embodiment of the present disclosure. Fig. 9 As shown, the image processing method 900 according to an embodiment of the present disclosure may start at step S902 and end at step S908.

[0120] like Fig. 9 As shown, the image processing method 900 according to an embodiment of the present disclosure may include an image generating step S904 and an image recognizing step S906.

[0121] In the image generation step S904, multiple motion-related parameters may be set for the input data to change the input data, and a pre-trained diffusion generation model may be used to generate a second image based on the input data in the change process. The multiple motion-related parameters may include displacement, velocity, and acceleration. For example, the image generation step S904 may be performed by the image generation unit 602, and thus will not be described below.

[0122] In the image recognition step S906, the object (eg, face, object, animal, etc.) involved in the second image generated in the image generation step S904 may be recognized. For example, the image recognition step S906 may be performed by the above-mentioned image recognition unit 604, and thus will not be described below.

[0123] It should be pointed out that although the functional configuration and operation of the image processing device and method and the model training device and method according to the embodiments of the present disclosure are described above, this is only an example and not a limitation, and those skilled in the art may modify the above embodiments according to the principles of the present disclosure, for example, the functional modules in each embodiment may be added, deleted or combined, and such modifications shall fall within the scope of the present disclosure.

[0124] In addition, it should be pointed out that the method embodiment here corresponds to the above-mentioned device embodiment. Therefore, for the contents not described in detail in the method embodiment, please refer to the description of the corresponding parts in the device embodiment, and the description will not be repeated here.

[0125] It should be understood that the machine-executable instructions in the storage medium and program products according to the embodiments of the present disclosure can also be configured to execute the above-mentioned image processing method and model training method. Therefore, the contents not described in detail here can refer to the description of the previous corresponding parts and will not be repeated here.

[0126] Accordingly, the storage medium for carrying the program product including the machine executable instructions is also included in the disclosure of the present invention, including but not limited to a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0127] In addition, it should be noted that the above series of processes and systems can also be implemented by software and / or firmware. In the case of being implemented by software and / or firmware, the data is transmitted from a storage medium or a network to a computer having a dedicated hardware structure, such as Fig.10 The general-purpose personal computer 1000 shown has programs constituting the software installed therein, and when the various programs are installed therein, the computer can execute various functions and the like.

[0128] exist Fig.10 In the embodiment, a central processing unit (CPU) 1001 executes various processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003. In the RAM 1003, data required when the CPU 1001 executes various processes and the like is also stored as needed.

[0129] The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. To the bus 1004, an input / output interface 1005 is also connected.

[0130] The following components are connected to the input / output interface 1005: an input device 1006 including a keyboard, a mouse, etc.; an output device 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage device 1008 including a hard disk, etc.; and a communication device 1009 including a network interface card such as a LAN card, a modem, etc. The communication device 1009 performs communication processing via a network such as the Internet.

[0131] A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage device 1008 as needed.

[0132] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1011 .

[0133] It should be understood by those skilled in the art that such storage media is not limited to Fig.10The removable medium 1011 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1011 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 1002, a hard disk included in the storage device 1008, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0134] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the present disclosure is certainly not limited to the above examples. Those skilled in the art may obtain various changes and modifications within the scope of the appended claims, and it should be understood that these changes and modifications will naturally fall within the technical scope of the present disclosure.

[0135] For example, a plurality of functions included in one unit in the above embodiments may be implemented by separate devices. Alternatively, a plurality of functions implemented by a plurality of units in the above embodiments may be implemented by separate devices, respectively. In addition, one of the above functions may be implemented by a plurality of units. Needless to say, such a configuration is included in the technical scope of the present disclosure.

[0136] In this specification, the steps described in the flowchart include not only the processing performed in time series in the order described, but also the processing performed in parallel or individually rather than necessarily in time series. In addition, even in the steps processed in time series, it goes without saying that the order can be appropriately changed.

[0137] Additionally, the technology according to the present disclosure may also be configured as follows.

[0138] Technical Solution 1. An image processing device, comprising:

[0139] an image generating unit configured to set a plurality of motion-related parameters for input data to change the input data, and to generate a second image based on the input data in the changing process using a pre-trained diffusion generation model,

[0140] Wherein, the multiple motion-related parameters include displacement, velocity and acceleration,

[0141] The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel of each original training image in a training image set to change the pixel, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model.

[0142] Technical Solution 2. According to the image processing device described in Technical Solution 1, for the multiple motion-related parameters, the image generation unit only applies random noise to the acceleration during the change process.

[0143] Technical Solution 3. An image processing device according to Technical Solution 1, wherein generating the second image comprises: generating the second image based on input data in a changing process under the guidance of the first image.

[0144] Technical Solution 4. The image processing device according to any one of Technical Solutions 1 to 3 further includes an image recognition unit configured to recognize the object involved in the second image.

[0145] Technical solution 5. A model training device, comprising a processing unit, configured to:

[0146] Setting a plurality of motion-related parameters for each pixel of each original training image in the training image set to change the pixel, and training a score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain a trained diffusion generation model for generating images,

[0147] Wherein, the multiple motion-related parameters include displacement, velocity and acceleration.

[0148] Technical Solution 6. According to the model training device of Technical Solution 5, the processing unit trains the score-based diffusion generation model based on the difference between the changed training images of the multiple stages and the original training images.

[0149] Technical Solution 7. According to the model training device described in Technical Solution 5 or 6, for the multiple motion-related parameters, the processing unit only applies random noise to the acceleration during the change process.

[0150] Technical solution 8. An image processing method, comprising:

[0151] Setting a plurality of motion-related parameters for input data to change the input data, and using a pre-trained diffusion generation model to generate a second image based on the input data in the change process,

[0152] Wherein, the multiple motion-related parameters include displacement, velocity and acceleration,

[0153] The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel of each original training image in a training image set to change the pixel, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model.

[0154] Technical Solution 9. The image processing method according to Technical Solution 8 also includes: applying random noise to the acceleration during the change process.

[0155] Technical Solution 10. An image processing method according to Technical Solution 8 or 9, wherein generating the second image comprises: generating the second image based on input data in a changing process under the guidance of the first image.

[0156] Technical Solution 11. The image processing method according to Technical Solution 8 or 9 also includes: identifying the object involved in the second image.

[0157] Technical solution 12. A computer-readable storage medium storing a program, wherein when the program is executed by a computer, the computer performs the image processing method according to any one of technical solutions 8 to 11.

Claims

1. An image processing device, comprising: an image generating unit configured to set a plurality of motion-related parameters for input data to change the input data, and to generate a second image based on the input data in the changing process using a pre-trained diffusion generation model, Wherein, the multiple motion-related parameters include displacement, velocity and acceleration, The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel of each original training image in a training image set to change the pixel, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model. 2 . The image processing device according to claim 1 , wherein, for the plurality of motion-related parameters, the image generation unit applies random noise only to the acceleration during the change process.

3. The image processing apparatus according to claim 1, wherein generating the second image comprises: The second image is generated based on the input data in the changing process under the guidance of the first image. 4 . The image processing apparatus according to claim 1 , further comprising an image recognition unit configured to recognize an object to which the second image relates. 5 .

5. A model training device, comprising a processing unit, configured to: Setting a plurality of motion-related parameters for each pixel of each original training image in the training image set to change the pixel, and training a score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain a trained diffusion generation model for generating images, in, The plurality of motion-related parameters include displacement, velocity and acceleration. 6 . The model training apparatus according to claim 5 , wherein the processing unit trains the score-based diffusion generation model based on the difference between the changed training images of the multiple stages and the original training images.

7. According to the model training device of claim 5 or 6, for the multiple motion-related parameters, the processing unit only applies random noise to the acceleration during the change process.

8. An image processing method, comprising: Setting a plurality of motion-related parameters for input data to change the input data, and using a pre-trained diffusion generation model to generate a second image based on the input data in the change process, Wherein, the multiple motion-related parameters include displacement, velocity and acceleration, The pre-trained diffusion generation model is obtained by setting the plurality of motion-related parameters for each pixel of each original training image in a training image set to change the pixel, and training the score-based diffusion generation model based on the changed training images at multiple stages in the change process to obtain the pre-trained diffusion generation model.

9. The image processing method according to claim 8, further comprising: Random noise is applied to the acceleration during the changing process. 10 . A computer-readable storage medium storing instructions, wherein when the instructions are executed by a computer, the computer executes the image processing method according to claim 8 or 9.

Images