Image extrapolation method and device, electronic device and storage medium

Abstract

The present application provides an image extrapolation method and device, an electronic device and a storage medium, the method comprising: obtaining the semantics of an image to be extrapolated, and expanding the semantics to obtain expanded semantics; obtaining a first initial latent space noise point, using the image to be extrapolated and the expanded semantics as restriction conditions of a pre-trained latent space diffusion model, and inputting the first initial latent space noise point into the latent space diffusion model to obtain an extrapolated image, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using the semantics of a sample image, a cropped image and a sample pattern as training samples, and obtaining a cropped image by cropping the sample image, so as to obtain an extrapolated image that is more natural and has richer content.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image extrapolation method and device, electronic equipment and storage medium. Background Art

[0002] Image extrapolation refers to expanding the content of an image by generating the peripheral area of ​​the image based on the existing image. In recent years, with the rapid development of artificial intelligence and deep learning technology, AIGC (artificial intelligence generated content) has made significant progress in the field of image generation. The explosive growth of computing power and the rapid development of latent space diffusion models have enabled image extrapolation technology to develop from traditional image splicing methods to automatic generation of image generation models.

[0003] Image extrapolation technology has a wide range of applications. It can be used in the restoration of old photos by filling in the missing parts to restore them to their complete state. It can also be used in artistic creation by expanding existing images to generate new visual effects or ideas.

[0004] The current mainstream extrapolation technology is usually based on the existing latent space diffusion model to train an image extrapolation model. This method has the following limitations:

[0005] 1. The generated image cannot be seamlessly connected with the original image, and it is easy to appear unnatural and unreal.

[0006] 2. Because the image semantics cannot be fully understood, the expanded image content is often simple and monotonous, and the main elements of the image are repeatedly stacked, which cannot generate complex scene elements.

[0007] Therefore, how to provide an image extrapolation solution to generate natural and content-rich extrapolated images has become a technical problem that needs to be solved urgently. Summary of the invention

[0008] In view of the technical problems existing in the prior art, the embodiments of the present application provide an image extrapolation method and device, an electronic device and a storage medium.

[0009] In a first aspect, an embodiment of the present application provides an image extrapolation method, comprising:

[0010] Acquiring semantics of the image to be extrapolated, and expanding the semantics to obtain expanded semantics;

[0011] A first initial latent space noise point is obtained, and an extrapolated image is obtained by using the image to be extrapolated and the extended semantics as constraints of a pre-trained latent space diffusion model and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using a sample image, a cropped image, and the semantics of a sample pattern as training samples, and the cropped image is obtained by cropping the sample image.

[0012] In a second aspect, the embodiment of the present application further provides an image extrapolation device, comprising:

[0013] An expansion unit, used for acquiring the semantics of the image to be extrapolated, and expanding the semantics to obtain expanded semantics;

[0014] A processing unit is used to obtain a first initial latent space noise point, and obtain an extrapolated image by using the image to be extrapolated and the extended semantics as the restriction conditions of a pre-trained latent space diffusion model, and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using a sample image, a cropped image, and the semantics of a sample pattern as training samples, and the cropped image is obtained by cropping the sample image.

[0015] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image extrapolation method described in the first aspect are executed.

[0016] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image extrapolation method described in the first aspect.

[0017] In summary, the image extrapolation method and device, electronic device and storage medium provided in the embodiments of the present application expand the semantics of the extrapolated image and use the latent space diffusion model trained by using the semantics of sample images, cropped images and sample patterns as training samples to process the expanded semantics, so that a more natural and content-rich extrapolated image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of an embodiment of an image extrapolation method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the structure of an image extrapolation device according to an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0022] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0023] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0024] In the prior art, it is difficult to obtain additional content information by directly parsing the image semantics and extrapolating the image, so it is also difficult to obtain rich content in the expanded part of the image. Therefore, the applicant's expansion of image semantics is not performed in the image domain, but in the semantic domain: first, the image semantics is parsed, then the image semantics is expanded, and finally, based on the expanded semantics and with the original image as a constraint, a natural and content-rich high-quality expanded image is generated through a latent space diffusion model.

[0025] Specifically, refer to Figure 1 As shown, an embodiment of the present application provides a flow chart of an image extrapolation method, the image extrapolation method comprising:

[0026] S10, acquiring the semantics of the image to be extrapolated, and expanding the semantics to obtain expanded semantics;

[0027] In this embodiment, it should be noted that an existing semantic recognition model (such as a contrastive language-image pre-training model) can be used to extract semantics from the image to be extrapolated as the semantics of the image to be extrapolated. Afterwards, the semantics of the image to be extrapolated need to be expanded to obtain expanded semantics. It should be noted that the expanded semantics are different from the semantics of the image to be extrapolated. The expanded semantics have richer text content than the semantics of the image to be extrapolated. For example, the semantics of the image to be extrapolated is: a girl on the beach; the expanded semantics is: a girl on the beach, with a puppy playing around, and the sun shining on the sea level.

[0028] S11. Obtain a first initial latent space noise point, and obtain an extrapolated image by using the image to be extrapolated and the extended semantics as the constraints of a pre-trained latent space diffusion model, and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using sample images, cropped images, and semantics of sample patterns as training samples, and the cropped image is obtained by cropping the sample image.

[0029] In this embodiment, it should be noted that the first initial latent space noise can be a randomly generated latent space noise or a specified latent space noise. When using a pre-trained latent space diffusion model, the first initial latent space noise and the spliced ​​data can be input into the latent space diffusion model to obtain the latent space noise output by the latent space diffusion model, and then the latent space noise output by the latent space diffusion model can be decoded to obtain the extrapolated image, wherein the spliced ​​data is obtained by splicing the first encoded data and the second encoded data, the first encoded data is obtained by encoding the extrapolated image using an encoder, and the second encoded data is obtained by encoding the extended semantics using an encoder. In addition, it should be noted that the latent space diffusion model can be trained using the semantics of the sample image, the cropped image and the sample pattern as training samples, that is, each training sample includes a sample image, a cropped image and the semantics of the sample pattern, wherein the cropped image is obtained by cropping the sample pattern, that is, the cropped image is a part of the sample pattern.

[0030] The image extrapolation method provided in the embodiment of the present application expands the semantics of the extrapolated image and uses a latent space diffusion model trained using the semantics of sample images, cropped images and sample patterns as training samples to process the expanded semantics, thereby obtaining an extrapolated image that is more natural and has richer content.

[0031] Based on the foregoing method embodiment, the method may further include:

[0032] Acquire a first training sample, and use the first training sample to train a latent space diffusion model to obtain the pre-trained latent space diffusion model, wherein the first training sample includes a plurality of first sample data;

[0033] The obtaining of the first training sample may include:

[0034] Obtaining a second initial latent space noise point, obtaining a first image by inputting the second initial latent space noise point into a latent space diffusion model, and cropping the first image to obtain a second image;

[0035] The semantics of the first image is obtained, and first sample data is constructed based on the second initial latent space noise, the second image, the semantics of the first image, and the first image.

[0036] In this embodiment, it should be noted that in order to enable the latent space diffusion model to generate the desired extrapolated image according to the image to be extrapolated and the extended semantics, the latent space diffusion model needs to be tuned. The second initial latent space noise point can be a randomly generated latent space noise point or a specified latent space noise point. When the first image is cropped, the left area and the right area of ​​the first image can be cropped to obtain the second image (for example, the first image is cropped into 3 parts on average, and the middle part is retained). Each first sample data includes four parts of data: the second initial latent space noise point, the second image, the semantics of the first image, and the first image. When the first sample data is used to train the latent space diffusion model, the semantics of the second image and the first image are respectively encoded and spliced ​​as the input of the latent space diffusion model by using the second initial latent space noise point as the initial latent space noise point, and the third image with the same size as the first image is generated. The similarity between the first image and the third image is measured by the loss function, and the parameters of the latent space diffusion model are tuned by gradient descent.

[0037] Based on the foregoing method embodiment, the expanding the semantics to obtain the expanded semantics may include:

[0038] The semantics is input into a pre-trained large language model to obtain extended semantics.

[0039] Based on the above method embodiment, before inputting the semantics into a pre-trained large language model to obtain the expanded semantics, the method may further include:

[0040] A second training sample is obtained, and the large language model is trained using the second training sample to obtain the pre-trained large language model, wherein the second training sample includes a plurality of second sample data, and the second sample data includes semantics and extended semantics.

[0041] In this embodiment, it should be noted that in order to expand the semantics, a large language model needs to be used for few-sample learning. Specifically, some simple semantics need to be prepared first, and then the expanded semantics of these semantics are manually prepared. For example, the semantics is: calm sea surface; the expanded semantics is: on the calm sea surface, a sailboat is sailing, and a group of seagulls are flying in the blue sky and white clouds. After that, a small amount of (semantics, expanded semantics) data is used to train the large language model. After the training is completed, the large language model can be used to expand the semantics of the extrapolated image to obtain the expected expanded semantics.

[0042] In this embodiment, by performing few-shot learning on a large language model and tuning a latent space diffusion model based on specific training samples, and combining the capabilities of the large language model and the latent space diffusion model, an image extrapolation method can be implemented, which can generate rich and natural extrapolated images.

[0043] Reference Figure 2 FIG. 1 is a schematic diagram of a structure of an image extrapolation device provided in an embodiment of the present application, the device comprising:

[0044] An expansion unit 20, used to obtain the semantics of the image to be extrapolated, and expand the semantics to obtain expanded semantics;

[0045] The processing unit 21 is used to obtain a first initial latent space noise point, and obtain an extrapolated image by using the image to be extrapolated and the extended semantics as the restriction conditions of a pre-trained latent space diffusion model, and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using the semantics of the sample image, the cropped image and the sample pattern as training samples, and the cropped image is obtained by cropping the sample image.

[0046] The image extrapolation device provided in the embodiment of the present application expands the semantics of the extrapolated image and uses a latent space diffusion model trained using the semantics of sample images, cropped images and sample patterns as training samples to process the expanded semantics, thereby obtaining a more natural and content-rich extrapolated image.

[0047] Based on the above device embodiment, the device may further include:

[0048] A first training unit is used to obtain a first training sample, and use the first training sample to train the latent space diffusion model to obtain the pre-trained latent space diffusion model, wherein the first training sample includes a plurality of first sample data;

[0049] The obtaining of the first training sample may include:

[0050] Obtaining a second initial latent space noise point, obtaining a first image by inputting the second initial latent space noise point into a latent space diffusion model, and cropping the first image to obtain a second image;

[0051] The semantics of the first image is obtained, and first sample data is constructed based on the second initial latent space noise, the second image, the semantics of the first image, and the first image.

[0052] Based on the above device embodiment, the expansion unit can be used for:

[0053] The semantics is input into a pre-trained large language model to obtain extended semantics.

[0054] Based on the above device embodiment, the device may further include:

[0055] The second training unit is used to obtain a second training sample before the expansion unit inputs the semantics into a pre-trained large language model to obtain the expanded semantics, and train the large language model using the second training sample to obtain the pre-trained large language model, wherein the second training sample includes multiple second sample data, and the second sample data includes semantics and expanded semantics.

[0056] The image extrapolation device provided in the embodiment of the present application has an implementation process that is consistent with the image extrapolation method provided in the embodiment of the present application, and the effect that can be achieved is also the same as the image extrapolation method provided in the embodiment of the present application, which will not be repeated here.

[0057] like Figure 3 As shown, an electronic device provided by an embodiment of the present application includes: a processor 30, a memory 31 and a bus 32, wherein the memory 31 stores machine-readable instructions executable by the processor 30, and when the electronic device is running, the processor 30 communicates with the memory 31 through the bus 32, and the processor 30 executes the machine-readable instructions to perform the steps of the above-mentioned image extrapolation method.

[0058] Specifically, the memory 31 and the processor 30 can be general-purpose memories and processors, which are not specifically limited here. When the processor 30 runs the computer program stored in the memory 31, the image extrapolation method can be executed.

[0059] Corresponding to the above-mentioned image extrapolation method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image extrapolation method are executed.

[0060] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image extrapolation method, characterized in that: include: Acquiring the semantics of the image to be extrapolated, and expanding the semantics to obtain expanded semantics; A first initial latent space noise point is obtained, and an extrapolated image is obtained by using the image to be extrapolated and the extended semantics as constraint conditions of a pre-trained latent space diffusion model and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using a sample image, a cropped image, and the semantics of the sample image as training samples, and the cropped image is obtained by cropping the sample image, and each training sample includes a sample image, a cropped image, and the semantics of the sample image.

2. The method according to claim 1, characterized in that Also includes: Acquire a first training sample, and use the first training sample to train a latent space diffusion model to obtain the pre-trained latent space diffusion model, wherein the first training sample includes a plurality of first sample data; The step of obtaining a first training sample includes: Obtaining a second initial latent space noise point, obtaining a first image by inputting the second initial latent space noise point into a latent space diffusion model, and cropping the first image to obtain a second image; The semantics of the first image is obtained, and first sample data is constructed based on the second initial latent space noise, the second image, the semantics of the first image, and the first image.

3. The method according to claim 1 or 2, characterized in that The step of expanding the semantics to obtain expanded semantics includes: The semantics is input into a pre-trained large language model to obtain extended semantics.

4. The method according to claim 3, characterized in that Before inputting the semantics into a pre-trained large language model to obtain the expanded semantics, the method further includes: A second training sample is obtained, and the large language model is trained using the second training sample to obtain the pre-trained large language model, wherein the second training sample includes a plurality of second sample data, and the second sample data includes semantics and extended semantics.

5. An image extrapolation device, characterized in that: include: An expansion unit, used for acquiring the semantics of the image to be extrapolated, and expanding the semantics to obtain expanded semantics; A processing unit is used to obtain a first initial latent space noise point, and obtain an extrapolated image by using the image to be extrapolated and the extended semantics as the restriction conditions of a pre-trained latent space diffusion model, and inputting the first initial latent space noise point into the latent space diffusion model, wherein the pre-trained latent space diffusion model is obtained by training the latent space diffusion model using a sample image, a cropped image, and the semantics of the sample image as training samples, and the cropped image is obtained by cropping the sample image, and each training sample includes a sample image, a cropped image, and the semantics of the sample image.

6. The device according to claim 5, characterized in that Also includes: A first training unit is used to obtain a first training sample, and use the first training sample to train the latent space diffusion model to obtain the pre-trained latent space diffusion model, wherein the first training sample includes a plurality of first sample data; The step of obtaining a first training sample includes: Obtaining a second initial latent space noise point, obtaining a first image by inputting the second initial latent space noise point into a latent space diffusion model, and cropping the first image to obtain a second image; The semantics of the first image is obtained, and first sample data is constructed based on the second initial latent space noise, the second image, the semantics of the first image, and the first image.

7. The device according to claim 5 or 6, characterized in that The expansion unit is used for: The semantics is input into a pre-trained large language model to obtain extended semantics.

8. The device according to claim 7, characterized in that Also includes: The second training unit is used to obtain a second training sample before the expansion unit inputs the semantics into a pre-trained large language model to obtain the expanded semantics, and train the large language model using the second training sample to obtain the pre-trained large language model, wherein the second training sample includes multiple second sample data, and the second sample data includes semantics and expanded semantics.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image extrapolation method according to any one of claims 1 to 4 are executed.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image extrapolation method according to any one of claims 1 to 4.

Images