Image processing method and device, electronic device and storage medium

By using frame sequence encoding and latent space noise map as input, a set of consistent continuous images is generated using the latent space motion transformation mapping module. This solves the problems of high training difficulty and high cost in generating coherent and consistent images in existing technologies, and realizes low-cost continuous image generation.

CN118537416BActive Publication Date: 2024-11-01HAIMA CLOUD TIANJIN INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410974741.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-01
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

Existing technologies face challenges in generating continuous images with coherence and consistency, including high training difficulty, high cost, or limited generation capabilities.

Method used

By setting the frame number t for encoding, and inputting the latent space noise map and the encoded frame number into the image generation model, a set of consistent continuous images is generated using the latent space motion transformation mapping module and the latent space diffusion model.

Benefits of technology

It generates a set of consistent, continuous images at extremely low cost, reducing training difficulty and cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118537416B_ABST
    Figure CN118537416B_ABST
Patent Text Reader

Abstract

The present application provides an image processing method and device, an electronic device and a storage medium. The method includes: setting a frame number t, encoding the frame number t to obtain the encoded frame number t; inputting a latent space noise map p and the encoded frame number t into an image generation model to generate a target image. A set of consistent continuous images can be generated at an extremely low cost using a latent space noise map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method and device, electronic equipment and storage medium. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technologies, AIGC (artificial intelligence generated content) has made significant progress in the field of image generation. The explosive growth of computing power has made it possible to efficiently generate high-quality and highly realistic static images. However, generating coherent and consistent continuous images still faces many challenges. Consistent image generation technology must not only be able to generate a single image but also ensure temporal and spatial consistency. This is particularly important for animation, film, and other applications requiring dynamic visual effects. Consistent image generation requires smooth transitions in image content, consistent style, and continuity of details, which places higher demands on the model's understanding and processing capabilities.

[0003] Currently, there are two main approaches to achieving consistent image generation: 1. Training a video generation model; 2. Integrating a motion modeling module into an image diffusion model. The first approach is extremely difficult to train a video model and requires high computing power and training data sets, resulting in high costs. While the second approach can reuse a pre-trained image diffusion model, the simultaneous initialization and denoising of multiple frames involved in the image generation process limits the length of the generated video, limiting its generation capabilities. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide an image processing method and device, an electronic device, and a storage medium, which can generate a set of consistent continuous images at an extremely low cost using a latent space noise map.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Set the frame number t, and encode the frame number t to obtain the encoded frame number t;

[0007] A latent space noise map p and the encoded frame number t are input into the image generation model to generate a target image, wherein the image generation model includes a latent space motion transformation mapping module and a latent space diffusion model, the output of the latent space motion transformation mapping module is input into the latent space diffusion model, the latent space motion transformation mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalization layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, and the third module includes a second group of normalization layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence.

[0008] In a second aspect, an embodiment of the present application further provides an image processing device, comprising:

[0009] The encoding unit is used to set the frame sequence number t and encode the frame sequence number t to obtain the encoded frame sequence number t;

[0010] The generation unit is used to input a latent space noise map p and an encoded frame number t into an image generation model to generate a target image, wherein the image generation model includes a latent space motion transformation mapping module and a latent space diffusion model, the output of the latent space motion transformation mapping module is input into the latent space diffusion model, the latent space motion transformation mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalization layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, and the third module includes a second group of normalization layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence.

[0011] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described in the first aspect are executed.

[0012] In a fourth aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the image processing method described in the first aspect.

[0013] In summary, the image processing method and device, electronic device and storage medium provided by the embodiments of the present application set a frame number t, encode the frame number t to obtain the encoded frame number t, and input a latent space noise map p and the encoded frame number t into an image generation model including a latent space motion transformation mapping module to generate a target image. In the entire scheme, by using the latent space motion transformation mapping module and setting the frame number t, a group of consistent continuous images can be generated at an extremely low cost using a latent space noise map. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of an image processing method according to an embodiment of the present application;

[0015] Figure 2 for Figure 1 The network architecture diagram of the latent space motion transformation mapping module used in;

[0016] Figure 3 A schematic structural diagram of an image processing device according to an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0019] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0021] Reference Figure 1 FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application, wherein the image processing method includes:

[0022] S10, setting a frame sequence number t, and encoding the frame sequence number t to obtain an encoded frame sequence number t;

[0023] In this embodiment, it should be noted that the frame sequence number t may include at least one frame sequence number, and the frame sequence number t may be encoded using an existing encoding method, such as sin encoding, cos encoding, and the like.

[0024] S11. Input a latent space noise map p and the encoded frame number t into the image generation model to generate a target image, wherein the image generation model includes a latent space motion transformation mapping module and a latent space diffusion model, the output of the latent space motion transformation mapping module is input into the latent space diffusion model, the latent space motion transformation mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalization layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, and the third module includes a second group of normalization layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence.

[0025] In this embodiment, it should be noted that if the frame number t is a single frame number, the frame number t can be set to a different value each time, ultimately resulting in different target images through steps S10 and S11, with these target images forming a set of consistent continuous images. Alternatively, an additional processing module can be added to the image generation model to process the encoded frame number t into multiple values ​​(e.g., encoded frame numbers corresponding to multiple different frame numbers). In this way, multiple target images can be obtained through processing in step S11, with these multiple target images forming a set of consistent continuous images. Furthermore, if the frame number t is at least two frame numbers, then a latent space noise map p and the encoded frame number t can be input into the image generation model to generate multiple target images, with these multiple target images forming a set of consistent continuous images.

[0026] The image processing method provided in the embodiment of the present application sets a frame number t, encodes the frame number t to obtain the encoded frame number t, and inputs a latent space noise map p and the encoded frame number t into an image generation model including a latent space motion transformation mapping module to generate a target image. In the entire scheme, by using the latent space motion transformation mapping module and setting the frame number t, a set of consistent continuous images can be generated at an extremely low cost using a latent space noise map.

[0027] like Figure 2 Shown Figure 1 The network architecture diagram of another latent space motion transformation mapping module used in, Figure 2 In the embodiment, the latent space motion transformation mapping module may further include: a fourth module, wherein the fourth module includes a third convolutional layer, the input of the fourth module is a latent space noise map, and the output of the third module and the output of the fourth module are summed to obtain the output of the latent space motion transformation mapping module.

[0028] Based on the aforementioned method embodiment, before inputting a latent space noise map p and the encoded frame sequence number t into the image generation model, the method may further include:

[0029] Divide a specified video into multiple images, input the multiple images into a latent space diffusion model to obtain multiple new images, and generate a training set based on the multiple new images, wherein the training set includes multiple training sample pairs, each training sample pair includes an input image, an encoded frame sequence number, and an output image;

[0030] The image generation model is trained using the training set, wherein during the training of the image generation model, parameters of the latent space diffusion model are frozen.

[0031] In this embodiment, it should be noted that when the specified video is segmented into multiple images, the segmentation can be performed at a fixed frame rate. Assuming that the multiple images segmented into the specified video are input into the latent space diffusion model, the multiple new images obtained include img(0) and img(2), then the training set generated based on the multiple new images can include a training sample pair (img(0), Encoder(2), img(2)), where img(0) represents the first frame image, img(2) represents the third frame image, and Encoder(2) represents the result of encoding frame number 2 (2 is the difference between frame number 2 corresponding to img(2) and frame number 0 corresponding to img(0)). Img(0) and Encoder(2) can be used as input data of the image generation model, and img(2) can be used as output data of the image generation model. It should be noted that the input data for the image generation model in a training sample pair may include the results of encoding at least two frame numbers, and the output data for the image generation model in the training sample pair may include at least two new images. In this case, each of the at least two frame numbers is the difference between the frame number corresponding to a new image in the training sample pair that serves as the output data of the image generation model and the frame number corresponding to the new image in the training sample pair that serves as the input data of the image generation model. During the image generation model training process, the parameters of the latent space diffusion model are frozen, that is, the parameters of the latent space diffusion model always remain unchanged at their initial values.

[0032] Based on the aforementioned method embodiment, when generating a target image, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module are zero.

[0033] In this embodiment, it should be noted that when generating the target image, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module need to be turned off, that is, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module are set to 0.

[0034] Reference Figure 3 FIG. 1 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, the device comprising:

[0035] The encoding unit 30 is used to set a frame sequence number t and encode the frame sequence number t to obtain an encoded frame sequence number t;

[0036] The generation unit 31 is used to input a latent space noise map p and an encoded frame number t into an image generation model to generate a target image, wherein the image generation model includes a latent space motion transformation mapping module and a latent space diffusion model, the output of the latent space motion transformation mapping module is input into the latent space diffusion model, the latent space motion transformation mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalization layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, and the third module includes a second group of normalization layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence.

[0037] The image processing device provided in the embodiment of the present application sets a frame number t, encodes the frame number t to obtain the encoded frame number t, and inputs a latent space noise map p and the encoded frame number t into an image generation model including a latent space motion transformation mapping module to generate a target image. In the entire scheme, by using the latent space motion transformation mapping module and setting the frame number t, a group of consistent continuous images can be generated at an extremely low cost using a latent space noise map.

[0038] Based on the aforementioned device embodiment, the latent space motion transformation mapping module may further include: a fourth module, wherein the fourth module includes a third convolutional layer, the input of the fourth module is a latent space noise map, and the output of the third module and the output of the fourth module are summed to obtain the output of the latent space motion transformation mapping module.

[0039] Based on the above device embodiment, the device may further include:

[0040] A training unit is configured to, before the generation unit operates, divide a specified video into multiple images, input the multiple images into a latent space diffusion model to obtain multiple new images, and generate a training set based on the multiple new images, wherein the training set includes multiple training sample pairs, each training sample pair including an input image, an encoded frame sequence number, and an output image; and train an image generation model using the training set, wherein the parameters of the latent space diffusion model are frozen during the training of the image generation model.

[0041] Based on the aforementioned device embodiment, when generating a target image, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module are zero.

[0042] The image processing device provided in the embodiment of the present application has an implementation process consistent with the image processing method provided in the embodiment of the present application, and the effect that can be achieved is also the same as the image processing method provided in the embodiment of the present application, which will not be repeated here.

[0043] Through the above solution, while the existing cloud application terminal's video frame network reception + decoding + rendering mode remains unchanged, a dynamic delay is added between network reception and decoding to smooth the decoding interval between the previous and next frames. This can optimize the decoding jams caused by the instantaneous reception of a large number of video frames due to network jitter and the decoding jams caused by insufficient terminal performance in a certain period of time, thereby optimizing the cloud application screen playback jams.

[0044] like Figure 4 As shown, an electronic device provided by an embodiment of the present application includes: a processor 40, a memory 41 and a bus 42, wherein the memory 41 stores machine-readable instructions executable by the processor 40. When the electronic device is running, the processor 40 communicates with the memory 41 through the bus 42, and the processor 40 executes the machine-readable instructions to perform the steps of the above-mentioned image processing method.

[0045] Specifically, the memory 41 and the processor 40 can be general-purpose memories and processors, which are not specifically limited here. When the processor 40 runs the computer program stored in the memory 41, the image processing method can be executed.

[0046] Corresponding to the above-mentioned image processing method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image processing method are executed.

[0047] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Set a frame number t, and encode the frame number t to obtain an encoded frame number t; A latent space noise map p and an encoded frame number t are input into an image generation model to generate a target image, wherein the image generation model includes a latent space motion transform mapping module and a latent space diffusion model, the output of the latent space motion transform mapping module is input into the latent space diffusion model, the latent space motion transform mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalized layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, the third module includes a second group of normalized layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence, and the target image is obtained by inputting the output of the third module into the latent space diffusion model.

2. The method according to claim 1, characterized in that The latent space motion transformation mapping module further includes: a fourth module, wherein the fourth module includes a third convolutional layer, and the input of the fourth module is a latent space noise map, The step of obtaining a target image by inputting the output of the third module into the latent space diffusion model comprises: The output of the third module and the output of the fourth module are summed to obtain the output of the latent space motion transformation mapping module, and the output of the latent space motion transformation mapping module is input into the latent space diffusion model to obtain the target image.

3. The method according to claim 1 or 2, characterized in that Before inputting a latent space noise map p and the encoded frame sequence number t into the image generation model, the method further includes: Divide a specified video into multiple images, input the multiple images into a latent space diffusion model to obtain multiple new images, and generate a training set based on the multiple new images, wherein the training set includes multiple training sample pairs, each training sample pair includes an input image, an encoded frame sequence number, and an output image; The image generation model is trained using the training set, wherein during the training of the image generation model, the parameters of the latent space diffusion model are frozen.

4. The method according to claim 1, characterized in that When generating the target image, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module are 0.

5. An image processing device, characterized in that: include: The encoding unit is used to set the frame sequence number t, and encode the frame sequence number t to obtain the encoded frame sequence number t; A generation unit is used to input a latent space noise map p and an encoded frame number t into an image generation model to generate a target image, wherein the image generation model includes a latent space motion transformation mapping module and a latent space diffusion model, the output of the latent space motion transformation mapping module is input into the latent space diffusion model, the latent space motion transformation mapping module includes a first module, a second module and a third module, the output of the first module and the output of the second module are summed and input into the third module, the input of the first module is the latent space noise map, the input of the second module is the encoded frame number, the first module includes a first group of normalized layers, a first activation function layer, a first Dropout layer and a first convolutional layer connected in sequence, the second module includes a second activation function layer and a fully connected layer connected in sequence, the third module includes a second group of normalized layers, a third activation function layer, a second Dropout layer and a second convolutional layer connected in sequence, and the target image is obtained by inputting the output of the third module into the latent space diffusion model.

6. The device according to claim 5, characterized in that The latent space motion transformation mapping module further includes: a fourth module, wherein the fourth module includes a third convolutional layer, and the input of the fourth module is a latent space noise map, The step of obtaining a target image by inputting the output of the third module into the latent space diffusion model comprises: The output of the third module and the output of the fourth module are summed to obtain the output of the latent space motion transformation mapping module, and the output of the latent space motion transformation mapping module is input into the latent space diffusion model to obtain the target image.

7. The device according to claim 5 or 6, characterized in that Also includes: A training unit is used to, before the generation unit works, divide a specified video into multiple images, input the multiple images into a latent space diffusion model to obtain multiple new images, and generate a training set based on the multiple new images, wherein the training set contains multiple training sample pairs, each training sample pair contains an input image, an encoded frame number and an output image; and use the training set to train the image generation model, wherein during the training of the image generation model, the parameters of the latent space diffusion model are frozen.

8. The device according to claim 5, characterized in that When generating the target image, the first Dropout layer and the second Dropout layer in the latent space motion transform mapping module are 0.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 4 are executed.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the image processing method as described in any one of claims 1 to 4.