Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, image processing and machine learning. The image processing method comprises the following steps: in response to an image expansion request, performing visual language recognition on a first original image to obtain an image description text; generating a second original image and a first mask image based on the first original image by using a target aspect ratio and a first target resolution included in the image expansion request; performing image generation based on the first mask image by taking the image description text and the second original image as semantic references to obtain a first target mask image; and carrying out image splicing on the first target mask image and the second original image to obtain a target extended image.

Description

[0001] The application number of this disclosure is 202310899530.3, the application date is July 20, 2023, and the priority of the invention patent application named "Image processing method, device, electronic device and storage medium" is claimed. The contents of the priority text are all cited in this disclosure. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision, image processing, and machine learning technology, and more specifically to an image processing method, device, electronic device, and storage medium. Background Art

[0003] Image extension technology is an image generation technology that aims to understand the semantic content and structure of an image and generate an extended part that is consistent with the style and content of the original image, so as to expand the boundary or canvas of the image and generate a larger or more complete image. It has wide applications in image editing, virtual reality and other fields. Summary of the invention

[0004] In view of this, the present disclosure provides an image processing method, apparatus, electronic device, readable storage medium, and computer program product.

[0005] One aspect of the present disclosure provides an image processing method, comprising: in response to an image extension request, performing visual language recognition on a first original image to obtain an image description text; using a target aspect ratio and a first target resolution included in the image extension request to generate a second original image and a first mask image based on the first original image; using the image description text and the second original image as semantic references, performing image generation based on the first mask image to obtain a first target mask image; and performing image stitching on the first target mask image and the second original image to obtain a target extended image.

[0006] Another aspect of the present disclosure provides an image processing method, including: in response to a first task start control being triggered, obtaining a first original image from the return result of a first image input control included in a display interface; performing visual language recognition on the above-mentioned first original image to obtain an image description text; based on the return results of each of a plurality of first configuration controls included in the above-mentioned display interface, obtaining a target aspect ratio and a first target resolution; using the above-mentioned target aspect ratio and the above-mentioned first target resolution, based on the above-mentioned first original image, generating a second original image and a first mask image; using the above-mentioned image description text and the above-mentioned second original image as semantic references, performing image generation based on the above-mentioned first mask image to obtain a first target mask image; and performing image stitching on the above-mentioned first target mask image and the above-mentioned second original image to obtain a target extended image.

[0007] Another aspect of the present disclosure provides an image processing device, including: a first recognition module, used to perform visual language recognition on a first original image in response to an image extension request to obtain an image description text; a first generation module, used to generate a second original image and a first mask image based on the first original image using a target aspect ratio and a first target resolution included in the image extension request; a first image generation module, used to perform image generation based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image; and a first stitching module, used to stitch the first target mask image with the second original image to obtain a target extended image.

[0008] Another aspect of the present disclosure provides an image processing device, including: a first acquisition module, used to acquire a first original image from the return result of a first image input control included in a display interface in response to a first task start control being triggered; a second recognition module, used to perform visual language recognition on the above-mentioned first original image to obtain an image description text; a second acquisition module, used to obtain a target aspect ratio and a first target resolution based on the return results of each of a plurality of first configuration controls included in the above-mentioned display interface; a second generation module, used to generate a second original image and a first mask image based on the above-mentioned first original image using the above-mentioned target aspect ratio and the above-mentioned first target resolution; a second image generation module, used to perform image generation based on the above-mentioned first mask image using the above-mentioned image description text and the above-mentioned second original image as semantic references to obtain a first target mask image; and a second stitching module, used to stitch the above-mentioned first target mask image with the above-mentioned second original image to obtain a target extended image.

[0009] Another aspect of the present disclosure provides a method for generating a dynamic image, comprising: in response to a second task start control being triggered, obtaining a prompt word text based on the return result of a prompt word input control included in a display interface; using the prompt word text as a semantic reference, performing image generation based on an initial template image to obtain a first template image; and generating a target dynamic image based on the first template image according to the return result of a dynamic image configuration control included in the display interface.

[0010] Another aspect of the present disclosure provides a dynamic image generation device, including: a third acquisition module, used to obtain a prompt word text based on the return result of the prompt word input control included in the display interface in response to the second task start control being triggered; a third image generation module, used to use the above prompt word text as a semantic reference, generate an image based on an initial template image, and obtain a first template image; and a dynamic image generation module, used to generate a target dynamic image based on the above first template image according to the return result of the dynamic image configuration control included in the above display interface.

[0011] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0012] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0013] Another aspect of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0016] Figure 1 An exemplary system architecture to which the image processing method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0017] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0018] Figure 3A The schematic diagram schematically shows the generation process of the second original image and the first mask image according to an embodiment of the present disclosure.

[0019] Figure 3B The following is a schematic diagram schematically showing a generation process of a second original image and a first mask image according to another embodiment of the present disclosure.

[0020] Figure 3C The following is a schematic diagram schematically showing a process of generating a second original image and a first mask image according to yet another embodiment of the present disclosure.

[0021] Figure 4 The following is a schematic diagram schematically showing the process of generating a target extended image.

[0022] Figure 5 The structural diagram of the image information creator according to the embodiment of the present disclosure is schematically shown.

[0023] Fig. 6A The structural diagram of the intermediate module according to the embodiment of the present disclosure is schematically shown.

[0024] Figure 6B The structural diagram of the intermediate module according to another embodiment of the present disclosure is schematically shown.

[0025] Figure 6C The schematic diagram shows the structure of a gated convolutional layer according to another embodiment of the present disclosure.

[0026] Figure 7 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0027] Fig. 8A A schematic diagram of a first original image according to another embodiment of the present disclosure is schematically shown.

[0028] Figure 8B A schematic diagram of an image after one expansion according to another embodiment of the present disclosure is schematically shown.

[0029] Figure 8C A schematic diagram of a target extended image according to another embodiment of the present disclosure is schematically shown.

[0030] Fig. 9 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0031] Fig. 10A A schematic diagram of a first display interface according to an embodiment of the present disclosure is schematically shown.

[0032] Fig. 10B A schematic diagram of a second display interface according to an embodiment of the present disclosure is schematically shown.

[0033] Fig. 10C A schematic diagram of a third display interface according to an embodiment of the present disclosure is schematically shown.

[0034] Fig.11 A block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure.

[0035] Fig.12 A block diagram of an image processing device according to another embodiment of the present disclosure is schematically shown.

[0036] Fig.13 The flowchart of the dynamic image generation method according to the embodiment of the present disclosure is schematically shown.

[0037] Fig.14 A schematic diagram of a fourth display interface according to an embodiment of the present disclosure is schematically shown.

[0038] Fig.15 A schematic diagram of a fifth display interface according to an embodiment of the present disclosure is schematically shown.

[0039] Fig.16 A schematic diagram of a fourth display interface according to another embodiment of the present disclosure is schematically shown.

[0040] Fig.17A A schematic diagram of a first template image according to an embodiment of the present disclosure is schematically shown.

[0041] Fig. 17B A schematic diagram of a second template image according to an embodiment of the present disclosure is schematically shown.

[0042] Fig.18A A schematic diagram of a target binarized image according to an embodiment of the present disclosure is schematically shown.

[0043] Fig.18B A schematic diagram of a third mask image according to an embodiment of the present disclosure is schematically shown.

[0044] Fig.19 A schematic diagram of a sixth display interface according to an embodiment of the present disclosure is schematically shown.

[0045] Fig. 20 A schematic diagram of a seventh display interface according to an embodiment of the present disclosure is schematically shown.

[0046] Fig.21 The block diagram of a dynamic image generating device according to an embodiment of the present disclosure is schematically shown.

[0047] Fig. 22 A block diagram of an electronic device suitable for implementing an image processing method or a dynamic image generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0049] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0050] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0051] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0052] The embodiments of the present disclosure provide an image processing method, device, electronic device and storage medium. The method includes: in response to an image extension request, performing visual language recognition on a first original image to obtain an image description text; using the target aspect ratio and the first target resolution included in the image extension request, based on the first original image, generating a second original image and a first mask image; using the image description text and the second original image as semantic references, performing image generation based on the first mask image to obtain a first target mask image; and performing image splicing on the first target mask image and the second original image to obtain a target extended image.

[0053] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0054] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0055] Figure 1 The exemplary system architecture to which the image processing method and apparatus according to the embodiments of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0056] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101 and 102 , a network 103 and a server 104 .

[0057] The terminal devices 101 and 102 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. Optionally, the resolutions of the display screens of the terminal devices 101 and 102 may be different.

[0058] The network 103 is used to provide a medium for communication links between the terminal devices 101, 102 and the server 104. The network 103 may include various connection types, such as wired and / or wireless communication links, and the like.

[0059] The server 104 can be a local server that provides various services, or it can be various cloud servers, which is not limited here.

[0060] It should be noted that the image processing method provided in the embodiment of the present disclosure can generally be executed by the server 104. Accordingly, the image processing device provided in the embodiment of the present disclosure can generally be set in the server 104. The image processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 104 and can communicate with the terminal devices 101, 102 and / or the server 104. Accordingly, the image processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 104 and can communicate with the terminal devices 101, 102 and / or the server 104. Alternatively, the image processing method provided in the embodiment of the present disclosure can also be executed by the terminal device 101 or 102, or it can also be executed by other terminal devices different from the terminal device 101 or 102. Accordingly, the image processing device provided in the embodiment of the present disclosure can also be set in the terminal device 101 or 102, or it can be set in other terminal devices different from the terminal device 101 or 102.

[0061] For example, the user may generate an image expansion request by operating the terminal device 101, and the image expansion request indicates that the first original image originally adapted to the display screen of the terminal device 101 is expanded into a target expanded image adapted to the display screen of the terminal device 102. The terminal device 101 may send the first original image and the image expansion request to the server 104 through the network 103, and the server 104 may respond to the image expansion request, execute the image processing method of the embodiment of the present disclosure to process the first original image, obtain the target expanded image, and send the target expanded image to the terminal device 102 through the network 103. After receiving the target expanded image, the terminal device 102 may display the target expanded image on its display screen.

[0062] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0063] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0064] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0065] In operation S210, in response to an image expansion request, visual language recognition is performed on the first original image to obtain an image description text.

[0066] In operation S220, a second original image and a first mask image are generated based on the first original image using the target aspect ratio and the first target resolution included in the image expansion request.

[0067] In operation S230, the image description text and the second original image are used as semantic references, and image generation is performed based on the first mask image to obtain a first target mask image.

[0068] In operation S240, the first target mask image is spliced ​​with the second original image to obtain a target extended image.

[0069] According to an embodiment of the present disclosure, the first original image may be the image on which the image is expanded, that is, the image style and content on which the image is expanded may be the image style and content of the first original image.

[0070] According to an embodiment of the present disclosure, performing visual language recognition on the first original image may be performing semantic analysis on the first original image, and the obtained image description text may contain semantic information of the first original image. Alternatively, performing visual language recognition on the first original image may be identifying various object items included in the first original image, and the obtained image description text may contain attribute information of the various object items.

[0071] According to an embodiment of the present disclosure, any vision-language pre-trained (VLP) model can be used to perform visual language recognition on the first original image. The vision-language pre-trained model can include a multimodal learning model based on images and texts, a multi-label classification model based on images and texts, etc., which is not limited here.

[0072] According to an embodiment of the present disclosure, the target aspect ratio and the first target resolution can be respectively expressed as the aspect ratio and resolution of the target extended image to be generated. The target aspect ratio and the first target resolution can be directly input into the terminal device by the user, or the user can also input relevant image extension information, and the target aspect ratio and the first target resolution can be calculated based on the relevant image extension information input into the terminal device by the user. For example, in a multi-screen extension scenario, the relevant image extension information input into the terminal device by the user may include the number of horizontal screens, the number of vertical screens, and the single-screen resolution. Based on the number of horizontal screens, the number of vertical screens, and the single-screen resolution, the target aspect ratio and the first target resolution can be calculated. Taking the number of horizontal screens as 2, the number of vertical screens as 3, and the single-screen resolution as 1920×1080 as an example, the first target resolution that can be calculated is (1920×2)×(1080×3)=3840×3840, and then according to the first target resolution, the aspect ratio can be calculated as 3840:3840=1:1.

[0073] According to an embodiment of the present disclosure, the second original image may be the first original image after resolution adjustment based on the first target resolution, that is, the long side resolution of the second original image may be equal to the long side resolution of the target extended image, or the short side resolution of the second original image may be equal to the short side resolution of the target extended image. The resolution scaling operation for the first original image may include a downsampling operation when performing resolution reduction, an interpolation operation when performing resolution enlargement, etc.

[0074] According to an embodiment of the present disclosure, the first mask image may be an image of an extended canvas portion. The canvas of the second original image is spliced ​​with the canvas of the first mask image to obtain a canvas of a target extended image to be generated.

[0075] According to an embodiment of the present disclosure, data of each pixel point in the image data of the generated first mask image may be any value.

[0076] According to an embodiment of the present disclosure, the image description text and the semantic information contained in the second original image can be used to guide the first mask image to generate an image, so that an image with the same content or style is generated in the first mask image part, thereby obtaining a first target mask image.

[0077] According to an embodiment of the present disclosure, the image of the extended canvas portion may be replaced by the first target mask image from the first mask image to achieve splicing of the first target mask image with the second original image.

[0078] According to an embodiment of the present disclosure, when performing image expansion, an image description text can be generated based on the semantic information of the first original image, a second original image can be generated based on the first original image, and a first mask image can be obtained by expansion. The image description text and the semantic information contained in the second original image are then used to guide the first mask image for image generation to obtain a first target mask image, so that the generated first target mask image can have the same or similar content and style as the first original image. The first target mask image can be spliced ​​with the second original image to obtain a target extended image, which can effectively improve the quality of the generated target extended image.

[0079] It should be noted that, unless it is clearly stated that there is a sequence of execution between different operations, or there is a sequence of execution between different operations in technical implementation, the operations shown in the flowchart in the embodiment of the present disclosure may be executed in any order, and multiple operations may be executed simultaneously. For example, after receiving an image expansion request, operation S210 may be executed first to generate an image description text, and then operation S220 may be executed to expand the first original image; or, operation S220 may be executed first to expand the first original image, and then operation S210 may be executed to generate an image description text; or, operation S210 and operation S220 may be executed simultaneously.

[0080] Reference below Figure 3A to Figure 3C , Figure 4 , Figure 5 , Figure 6A to Figure 6C , Figure 7 and Figure 8A to Figure 8C , combined with specific embodiments Figure 2 The method shown is further explained.

[0081] According to an embodiment of the present disclosure, a user may input a target aspect ratio and a first target resolution on a terminal device, and click a button control indicating “start generating”, so that the terminal device generates an image expansion request.

[0082] According to an embodiment of the present disclosure, in response to an image expansion request, the first original image can be processed using a visual language pre-trained model to obtain at least one object text; and then based on the at least one object text, an image description text is obtained, that is, text used to describe the first original image.

[0083] According to an embodiment of the present disclosure, the visual language pre-training model may be a CLIP (Contrastive Language-Image Pretraining) model. The CLIP model may realize semantic understanding and alignment between images and texts. The CLIP model may use a contrastive learning method and a large-scale image and text data set for pre-training. The CLIP model may learn to bring the embedding vectors of related image and text pairs close to each other in the semantic space, and separate the embedding vectors of unrelated image and text pairs from each other in the semantic space, so that the model has cross-modal capabilities and can link images and texts. When the CLIP model is applied, one or more matching texts, i.e., at least one object text, may be determined from a plurality of preset texts by matching the embedding vector of the input image with the respective embedding vectors of a plurality of preset texts. The at least one object text may be spliced ​​to obtain an image description text.

[0084] According to an embodiment of the present disclosure, the visual language pre-training model may also be a multi-label classification model, which may identify objects such as scenes, objects, and people in the first original image to obtain at least one object label. The name of each of the at least one object label may be used as at least one object text, and the at least one object text may be spliced ​​to obtain an image description text. Alternatively, the names of each of the at least one object label may be respectively filled into a text template to obtain at least one object text, and the at least one object text may be spliced ​​to obtain an image description text. Alternatively, the names of each of the at least one object label may be filled into a text template to directly obtain the image description text.

[0085] According to an embodiment of the present disclosure, as an optional implementation, for a specific scene, such as when the first original image is a poster-type, landscape-type, etc. image, or when the content to be expanded from the image is some specific scene, the developer can preset some keywords. For example, for a poster-type image, its preset keywords may include poster, gradient background, widescreen, illustration, vector painting, etc. For a landscape-type image, its preset keywords may include realistic style, film texture, photo-like painting, high quality, etc. Before generating an image extension request, a user may select an image extension keyword from the preset keywords, and accordingly, the image extension request may include the image extension keyword. When generating an image description text, an image description text may be obtained based on at least one object text and an image extension keyword. Specifically, at least one object text and an image extension keyword may be spliced ​​to obtain an image description text. Alternatively, the name of each of at least one object tag and the image extension keyword may be respectively filled into a text template to obtain at least one object text, and the at least one object text may be spliced ​​to obtain an image description text. Alternatively, the name of each of at least one object tag and the image extension keyword may be filled into a text template to directly obtain the image description text.

[0086] According to an embodiment of the present disclosure, while generating the image description text, the first original image can be expanded. The image expansion can be performed directly on the basis of the first original image. Specifically, the target aspect ratio and the first target resolution included in the image expansion request are used to generate the second original image and the first mask image based on the first original image. The following operations can be included:

[0087] Based on the target aspect ratio and the first target resolution, image expansion is performed with at least one side of the first original image as a starting point to obtain a first extended image; and a second original image and a first mask image are determined from the first extended image.

[0088] According to an embodiment of the present disclosure, image expansion is performed with at least one side of the first original image as a starting point, which may refer to image expansion with at least one side of the first original image as a common side. For example, the first original image may be a rectangular image surrounded by side A, side B, side C, and side D. When image expansion is performed with side A as a starting point, side B may be extended from the intersection of side B and side A, and the extended portion may be regarded as side B′, and side D may be extended from the intersection of side D and side A, and the extended portion may be regarded as side D′. The extended end points of side B′ and side D′ are connected to obtain a new side, which may be regarded as side A′. After image expansion is performed with side A as a starting point, the first extended image obtained may be composed of side A′, side B, side B′, side C, side D, and side D′, wherein side A′, side B′, side A, and side D′ constitute the first mask image.

[0089] According to an embodiment of the present disclosure, the second original image determined from the first extended image is the first original image.

[0090] Figure 3A The schematic diagram schematically shows the generation process of the second original image and the first mask image according to an embodiment of the present disclosure.

[0091] like Figure 3A As shown, the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, and the first target resolution may be 2048×2048. Based on the first target resolution and the target aspect ratio, the first original image 301 may be image expanded to obtain a first expanded image 302 with a resolution of 2048×2048. The second original image 303 that can be determined in the first expanded image 302 is the first original image 301, and the portion of the first expanded image 302 other than the first original image 301 may be the first mask image 304.

[0092] According to an embodiment of the present disclosure, a method of directly performing image expansion on the basis of the first original image is adopted. When the first target resolution is much larger than the original resolution, the second original image obtained after scaling has a larger size. Image expansion based on the second original image will consume more computing resources. Therefore, as an optional implementation method, in order to reduce the size of the image during calculation and improve the computing efficiency, the first original image can be first reduced in size, and after the image expansion is completed, the expanded image can be enlarged to the required resolution.

[0093] According to an embodiment of the present disclosure, when the first original image is reduced, the reduction process can be performed based on a preset resolution. The preset resolution can be expressed as a smaller resolution, and the preset resolution can only include the resolution of the long side or the resolution of the short side. The preset resolution can be set by a developer, or the preset resolution can also be obtained through a user's input operation, which is not limited here.

[0094] According to the embodiments of the present disclosure, in some embodiments, the preset resolution may be equal to the first target resolution. In this case, after the image expansion is completed, the desired image can be obtained. Specifically, using the target aspect ratio and the first target resolution included in the image expansion request, based on the first original image, generating the second original image and the first mask image may include the following operations:

[0095] Based on the original resolution and the preset resolution of the first original image, the first original image is scaled to obtain a third original image; based on the target aspect ratio, the image is expanded with at least one edge of the third original image as a starting point to obtain a second expanded image; when the first target resolution is equal to the preset resolution, the second original image and the first mask image are determined from the second expanded image.

[0096] According to an embodiment of the present disclosure, when the original resolution is greater than the preset resolution, the first original image may be downsampled to reduce the resolution of the first original image from the original resolution to the preset resolution. The downsampling process may include maximum pooling, average pooling, etc.

[0097] According to an embodiment of the present disclosure, when the original resolution is less than the preset resolution, the first original image may be interpolated, or the first original image may be processed using a super-resolution algorithm to increase the resolution of the first original image from the original resolution to the preset resolution. The interpolation process may include bilinear interpolation, nearest neighbor interpolation, and the like. The super-resolution algorithm may include an ESRGAN (Enhanced Super-Re solution Generative Adversarial Networks) method based on GAN (Generative Adversarial Networks), a transformer-based SwinIR (Image Restoration Using Swin Transformer) method, and the like.

[0098] According to an embodiment of the present disclosure, when the original resolution is equal to the preset resolution, the second original image may be determined to be the first original image.

[0099] According to an embodiment of the present disclosure, when performing scaling processing, the long side resolution and the short side resolution can be scaled proportionally to the long side resolution or the short side resolution represented by the preset resolution based on the long side or the short side of the first original image, so that the long side resolution of the third original image obtained after expansion is equal to the long side resolution represented by the preset resolution, or the short side resolution of the third original image is equal to the short side resolution represented by the preset resolution. For example, the resolution of the first original image can be 1024×512, and the preset resolution can be 2048×2048. Then, the resolution of the first original image can be enlarged by 2 times based on the long side of the first original image, and the resolution of the third original image obtained can be 2048×1024.

[0100] Figure 3BThe following is a schematic diagram schematically showing a generation process of a second original image and a first mask image according to another embodiment of the present disclosure.

[0101] like Figure 3B As shown, the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, and the first target resolution may be equal to the preset resolution, both of which are 2048×2048. By comparison, it can be seen that the original resolution of the first original image 301 is less than the preset resolution, so the first original image 301 may be scaled to obtain a third original image 305 with a resolution of 2048×1024. The image may be expanded based on the upper and lower boundaries of the third original image 305, with the third original image 305 as the center, according to the target aspect ratio, to obtain a second expanded image 306 with a resolution of 2048×2048. The second extended image 306 may include a first sub-mask image 3041 adjacent to the upper boundary of the third original image 305 and a second sub-mask image 3042 adjacent to the lower boundary of the third original image 305. The first sub-mask image 3041 and the second sub-mask image 3042 may constitute the first mask image 304. Since the first target resolution is equal to the preset resolution, the third original image 305 is the second original image 303.

[0102] According to an embodiment of the present disclosure, the first target resolution may generally be greater than the preset resolution. When the first target resolution is not equal to the preset resolution, the first extended image may be scaled based on the first target resolution and the preset resolution to obtain a third extended image; and the second original image and the first mask image may be determined from the third extended image.

[0103] According to an embodiment of the present disclosure, the scaling process for the first extended image may be performed using the same or similar method as that for scaling the first original image. The description of the scaling process for the first extended image refers specifically to the description of the scaling process for the first original image, which will not be repeated here.

[0104] Figure 3C The following is a schematic diagram schematically showing a process of generating a second original image and a first mask image according to yet another embodiment of the present disclosure.

[0105] like Figure 3CAs shown, the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, the first target resolution may be 2048×2048, and the preset resolution may indicate that the long side resolution is 512. By comparison, it is easy to see that the long side resolution in the original resolution is greater than the long side resolution indicated by the preset resolution, and the first target resolution is greater than the original resolution. Therefore, the first original image 301 may be scaled based on the preset resolution to obtain a third original image 305 with a resolution of 512×256. The image may be expanded based on the upper boundary of the third original image 305 with the third original image 305 as the center according to the target aspect ratio to obtain a second expanded image 306 with a resolution of 512×512. The second expanded image 306 may include the third original image 305 and an expanded portion 307. Since the first target resolution and the preset resolution are not equal, the second extended image 306 can be super-enlarged based on the first target resolution to obtain a third extended image 308 with a resolution of 2048×2048. After the super-enlargement, the third original image 305 can be enlarged to a second original image 303 with a resolution of 2048×1024, and the extended portion 307 can be enlarged to a first mask image 304 with a resolution of 2048×1024. In this way, the second original image 303 and the first mask image 304 can be determined from the third extended image 308.

[0106] According to an embodiment of the present disclosure, the second original image and the first mask image are obtained by reducing-expanding-enlarging. When the first target resolution is large, the size of the image during the image expansion operation can be effectively reduced, thereby reducing the consumption of computing resources and improving computing efficiency.

[0107] According to an embodiment of the present disclosure, after the second original image and the first mask image are generated, a diffusion model may be used to generate a target extended image.

[0108] Figure 4 The following is a schematic diagram schematically showing the process of generating a target extended image.

[0109] like Figure 4 As shown, the diffusion model may include a multimodal encoder 401 , an image information creator 402 , and a model decoder 403 .

[0110] According to an embodiment of the present disclosure, a multimodal encoder 401 may be used to encode the second original image 404 and the image description text 405 to obtain an image encoding feature 406 and a text encoding feature 407 .

[0111] According to an embodiment of the present disclosure, the multimodal encoder 401 can be used for encoding operations on the second original image 404 and the image description text 405, respectively. Alternatively, the multimodal encoder 401 can be divided into an image encoder 4011 and a text encoder 4012. The image encoder 4011 can encode the second original image 404 to obtain image encoding features 406. The text encoder 4012 can encode the image description text 405 to obtain text encoding features 407. The image encoder 4011 can be implemented, for example, using a variational autoencoder (VAE), and the text encoder 4012 can be implemented, for example, using a text encoder in a CLIP model.

[0112] According to an embodiment of the present disclosure, before the first mask image 408 is processed using a diffusion model, the first mask image 408 may be initialized using Gaussian noise to obtain a noise image 409. The image data of the noise image 409 may obey a normal distribution.

[0113] According to an embodiment of the present disclosure, the image information creator 402 can be used to process the noise image 409 based on the image coding feature 406 and the text coding feature 407 to obtain the target output feature 410 .

[0114] According to an embodiment of the present disclosure, the target output feature 410 can be decoded by using the model decoder 403 to obtain a first target mask image 411. The model decoder 403 may correspond to the multimodal encoder 401, or the model decoder 403 may correspond to the image encoder 4011 in the multimodal encoder 401. Through the model decoder 403, the image feature can be mapped back to the pixel space to obtain the first target mask image 411.

[0115] According to an embodiment of the present disclosure, the second original image 404 and the first target mask image 411 may be spliced ​​based on at least one edge of the second original image 404 as the starting point for image expansion to obtain a target extended image 412 to complete the image generation process.

[0116] According to an embodiment of the present disclosure, the image information creator 402 may be a U-Net neural network structure for predicting noise, that is, the image information creator 402 may be composed of an encoding module, a decoding module, and an intermediate module. The encoding module may include M encoding function blocks, and correspondingly, the decoding module may include M decoding function blocks.

[0117] According to an embodiment of the present disclosure, the image coding features and the text coding features are used as semantic benchmarks, and the image information creator is used to process the noisy image to obtain the target output features, which may include the following operations:

[0118] The encoding module is used to process the image encoding features, the text encoding features and the noise image to obtain the first noise image features output by each of the M encoding function blocks included in the encoding module, wherein M is a positive integer; the intermediate module is used to process the first noise image features output by the Mth encoding function block to obtain the second noise image features; and the decoding module is used to process the second noise image features, the first noise image features output by each of the M encoding function blocks, the image encoding features and the text encoding features to obtain the target output features.

[0119] The following takes M=4 as an example, that is, the encoding module includes 4 encoding functional blocks and the decoding module includes 4 decoding functional blocks, to illustrate the processing flow of the image information creator 402.

[0120] Figure 5 The structural diagram of the image information creator according to the embodiment of the present disclosure is schematically shown.

[0121] like Figure 5 As shown, the image information creator 402 may include multiple functional blocks from its input end to its output end, namely 4 encoding functional blocks 4021, 1 intermediate module 4022 and 4 decoding functional blocks 4023.

[0122] According to an embodiment of the present disclosure, the four encoding function blocks 4021 and the four decoding function blocks 4023 included in the image information creator 402 can be divided into four levels, and the encoding function blocks 4021 of each level can be jump-connected with the decoding function blocks 4023, and the first noise image features output by the encoding function blocks 4021 can be provided as input to the decoding function blocks 4023 at the same level. The first encoding function block 4021 can be at the same level as the fourth decoding function block 4023, the second encoding function block 4021 can be at the same level as the third decoding function block 4023, the third encoding function block 4021 can be at the same level as the second decoding function block 4023, and the fourth encoding function block 4021 can be at the same level as the first decoding function block 4023.

[0123] According to an embodiment of the present disclosure, the M encoding function blocks 4021 may be connected via a cross attention block 4024. Each cross attention block 4024 may process the input features of the cross attention block 4024 based on the image encoding features 406 and the text encoding features 407.

[0124] Specifically, using the encoding module to process the image encoding feature, the text encoding feature and the noise image to obtain the first noise image feature output by each of the M encoding function blocks included in the encoding module may include the following operations:

[0125] The j-th coding function block is used to process the input features of the j-th coding function block to obtain the first noise image features output by the j-th coding function block, wherein j is a positive integer less than or equal to M, and the input features of the first coding function block are represented as noise images; and when j is less than M, the cross-attention block between the j-th coding function block and the j+1-th coding function block is used to process the first noise image features, image coding features, and text coding features output by the j-th coding function block to obtain the input features of the j+1-th coding function block.

[0126] According to an embodiment of the present disclosure, specifically, the input feature of the first encoding function block 4021 may be image data of a noise image 409. The first encoding function block 4021 processes the image data of the noise image 409 to obtain a first noise image feature output by the first encoding function block 4021. The cross attention block 4024 located between the first encoding function block 4021 and the second encoding function block 4021 may process the first noise image feature output by the first encoding function block 4021 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the second encoding function block 4021.

[0127] The second encoding function block 4021 can process the input feature of the second encoding function block 4021 to obtain the first noise image feature output by the second encoding function block 4021. The cross attention block 4024 located between the second encoding function block 4021 and the third encoding function block 4021 can process the first noise image feature output by the second encoding function block 4021 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the third encoding function block 4021.

[0128] The third encoding function block 4021 can process the input feature of the third encoding function block 4021 to obtain the first noise image feature output by the third encoding function block 4021. The cross attention block 4024 located between the third encoding function block 4021 and the fourth encoding function block 4021 can process the first noise image feature output by the third encoding function block 4021 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the fourth encoding function block 4021.

[0129] The fourth encoding function block 4021 can process the input feature of the fourth encoding function block 4021 to obtain the first noise image feature output by the fourth encoding function block 4021.

[0130] According to an embodiment of the present disclosure, the intermediate module 4022 may process the first noise image feature output by the fourth encoding functional block 4021 to obtain a second noise image feature.

[0131] According to an embodiment of the present disclosure, the M decoding function blocks 4023 may be connected via a cross attention block 4024. Each cross attention block 4024 may process the input features of the cross attention block 4024 based on the image encoding features 406 and the text encoding features 407.

[0132] According to an embodiment of the present disclosure, using a decoding module to process the second noise image feature, the first noise image feature, the image coding feature, and the text coding feature output by each of the M coding function blocks to obtain the target output feature may include the following operations:

[0133] The kth decoding function block is used to process the input features of the kth decoding function block and the first noise image features output by the kth encoding function block to obtain the decoding features output by the kth decoding function block, wherein k is a positive integer less than or equal to M, and the input features of the first decoding function block are represented as the second noise image features; when k is less than M, the decoding features, image encoding features and text encoding features output by the kth decoding function block are processed by the cross attention block between the kth decoding function block and the k+1th decoding function block to obtain the input features of the k+1th decoding function block; and when k is equal to M, the target output feature is determined to be the decoding feature output by the kth decoding function block.

[0134] According to an embodiment of the present disclosure, specifically, the input feature of the first decoding function block 4023 may be a third image feature. The first decoding function block 4023 may process the third image feature and the first noise image feature output by the fourth encoding function block 4021 obtained via the jump connection to obtain the decoding feature output by the first decoding function block 4023. The cross attention block 4024 located between the first decoding function block 4023 and the second decoding function block 4023 may process the decoding feature output by the first decoding function block 4023 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the second decoding function block 4023.

[0135] The second decoding function block 4023 can process the input feature of the second decoding function block 4023 and the first noise image feature output by the third encoding function block 4021 obtained through the skip connection to obtain the decoding feature output by the second decoding function block 4023. The cross attention block 4024 located between the second decoding function block 4023 and the third decoding function block 4023 can process the decoding feature output by the second decoding function block 4023 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the third decoding function block 4023.

[0136] The third decoding function block 4023 can process the input feature of the third decoding function block 4023 and the first noise image feature output by the second encoding function block 4021 obtained through the skip connection to obtain the decoding feature output by the third decoding function block 4023. The cross attention block 4024 located between the third decoding function block 4023 and the fourth decoding function block 4023 can process the decoding feature output by the third decoding function block 4023 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the fourth decoding function block 4023.

[0137] The fourth decoding function block 4023 can process the input features of the fourth decoding function block 4023 and the first noise image features output by the first encoding function block 4021 obtained via the jump connection to obtain the decoding features output by the fourth decoding function block 4023, that is, the target output features 410.

[0138] According to an embodiment of the present disclosure, the encoding function block 4021, the intermediate module 4022 and the decoding function block 4023 included in the image information creator 402 can all be implemented using a residual block (ResBlock).

[0139] The following describes the processing flow of the residual block by taking the process in which the intermediate module 4022 processes the first noise image feature output by the Mth encoding function block 4021 to obtain the second noise image feature as an example.

[0140] Fig. 6A The structural diagram of the intermediate module according to the embodiment of the present disclosure is schematically shown.

[0141] like Fig. 6AAs shown, the intermediate module 4022 can be implemented using a residual block. For the first noise image feature output by the Mth encoding function block 4021 input to the intermediate module 4022, the first noise image feature output by the Mth encoding function block 4021 can be subjected to normalization processing, convolution processing, normalization processing, mask processing (dropout) and convolution processing in sequence to obtain a residual output feature, and the residual output feature can be added to the first noise image feature output by the Mth encoding function block 4021 after the convolution processing to obtain a second noise image feature.

[0142] According to an embodiment of the present disclosure, as an optional implementation, a gated convolution block (GnBlock) can be used to replace the residual module, that is, the encoding function block 4021, the intermediate module 4022 and the decoding function block 4023 can all be implemented using a gated convolution block.

[0143] The following describes the processing flow of the gated convolution block by taking the process in which the intermediate module 4022 processes the first noise image feature output by the Mth encoding function block 4021 to obtain the second noise image feature as an example.

[0144] Figure 6B The structural diagram of the intermediate module according to another embodiment of the present disclosure is schematically shown.

[0145] like Figure 6B As shown, the intermediate module 4022 can be implemented using a gated convolution block. For the first noise image feature output by the Mth encoding function block 4021 input to the intermediate module 4022, the first noise image feature output by the Mth encoding function block can be subjected to layer normalization processing to obtain a first sub-feature; the first sub-feature is processed using a gated convolution layer (GnConv) to obtain a second sub-feature; the second sub-feature is subjected to layer normalization processing to obtain a third sub-feature; the third sub-feature is processed using a perceptron to obtain a fourth sub-feature; and the second noise image feature is obtained based on the first noise image feature and the fourth sub-feature output by the Mth encoding function block.

[0146] According to an embodiment of the present disclosure, layer normalization processing may refer to normalizing the inputs of all neurons in batches, that is, making the data within the layer obey a normal distribution with a mean of 0 and a variance of 1, thereby speeding up processing efficiency and improving robustness to scaling changes during image processing.

[0147] According to an embodiment of the present disclosure, the perceptron may include multiple fully connected layers. Optionally, the multiple fully connected layers may be masked (dropout), that is, random positions and numbers of neurons in the multiple fully connected layers are controlled not to participate in the calculation of the output result.

[0148] Figure 6C The schematic diagram shows the structure of a gated convolutional layer according to another embodiment of the present disclosure.

[0149] like Figure 6C As shown, for the first sub-feature input into the gated convolution layer, the first sub-feature can be convolved to obtain the fifth sub-feature. The fifth sub-feature can be subjected to deep convolution (Deep-Wise Conv) to obtain a deep convolution feature. The gated convolution layer can include P dual-input gated processing units, where P can be a positive integer, one input of the gated processing unit can be a deep convolution feature, and the other input can be an output feature of the previous gated processing unit. Each gated processing unit can perform a dot multiplication on its two inputs and perform a convolution on the product to obtain its output feature. For example, for the pth gated processing unit, the pth gated processing unit can perform a dot multiplication on the deep convolution feature and the gated output feature output by the p-1th gated processing unit to obtain the sixth sub-feature of the pth gated processing unit, and then perform a convolution on the sixth sub-feature of the pth gated processing unit to obtain the gated output feature of the pth gated processing unit. Among them, p can be expressed as a positive integer, the gated output feature output by the 0th gated processing unit can be expressed as the fifth sub-feature, and the gated output feature of the Pth gated processing unit can be expressed as the second sub-feature.

[0150] According to an embodiment of the present disclosure, taking the gated processing units as 3 as an example, the deep convolution feature can be dot-multiplied with the fifth sub-feature, and the product can be convolved to obtain the sixth sub-feature. The deep convolution feature can be dot-multiplied with the sixth sub-feature, and the product can be convolved to obtain the seventh sub-feature. The deep convolution feature can be dot-multiplied with the seventh sub-feature, and the product can be convolved to obtain the second sub-feature of the output.

[0151] According to an embodiment of the present disclosure, the gated convolution layer in the gated convolution block can use gated convolution and recursion to achieve interaction between different spatial features and enhance feature expression, thereby improving the generation effect of the diffusion model.

[0152] According to an embodiment of the present disclosure, in some application scenarios, the first target resolution and the original resolution have a large difference, and when the difference is large, the content of the image portion obtained by image expansion may be greatly different from the first original image, and the content and style of the expanded image portion may be inconsistent with the first original image. Therefore, as an optional implementation, based on the first target resolution and the original resolution of the first original image, when it is determined that the preset conditions are met, the first original image and the image description text can be used as semantic references to perform N image expansions to obtain a target expanded image, where N is a positive integer.

[0153] According to an embodiment of the present disclosure, the preset condition may be that the ratio between the first target resolution and the original resolution of the first original image is less than a first threshold; or, the ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold.

[0154] According to an embodiment of the present disclosure, the ratio between the first target resolution and the original resolution of the first original image can be expressed as the ratio between the long side resolution in the first target resolution and the long side resolution in the original resolution, or can be expressed as the ratio between the short side resolution in the first target resolution and the short side resolution in the original resolution, or can be expressed as the smaller of the above two ratios, which is not limited here.

[0155] According to an embodiment of the present disclosure, the first threshold and the second threshold may be set according to a specific application scenario. For example, the first threshold may be set to 0.66, the second threshold may be set to 1.5, etc., which is not limited here.

[0156] According to an embodiment of the present disclosure, the size of N may include being determined based on the first target resolution and the original resolution. For example, the ratio between the first target resolution and the original resolution may be 2.25, and the resolution of each expansion may not exceed 50% of the image resolution. Then, the first original image may be expanded twice to obtain the target expanded image.

[0157] According to an embodiment of the present disclosure, a user may also configure the number of times of image extension by himself, and accordingly, the image extension request may include information of the number of times of image extension. As an optional implementation, the preset condition may also include that the number of times of image extension included in the image extension request is greater than 1. Accordingly, in this case, N represents the number of times of image extension.

[0158] According to an embodiment of the present disclosure, performing N times of image expansion using the first original image and the image description text as semantic references to obtain a target expanded image may include the following operations:

[0159] In the case where the first target resolution and the image resolution of the extended image after the i-1th image expansion are not equal, the second target resolution is determined based on the image resolution and the expansion ratio of the extended image after the i-1th image expansion, wherein i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution; using the target aspect ratio and the second target resolution, based on the extended image after the i-1th image expansion, a fourth original image and a second mask image are generated; using the image description text and the fourth original image as semantic references, image generation is performed based on the second mask image to obtain a second target mask image; and the fourth original image is image-stitched with the second target mask image to obtain the extended image after the i-th image expansion.

[0160] According to an embodiment of the present disclosure, when the first target resolution is equal to the image resolution of the extended image after the i-th image expansion, the extended image after the i-th image expansion can be determined as the target extended image. Alternatively, when the value of i is equal to N, the extended image after the i-th image expansion can be determined as the target extended image.

[0161] Figure 7 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0162] like Figure 7 As shown, the method includes operations S701 to S708.

[0163] In operation S701, an expansion magnification is determined based on a first target resolution, an original resolution of a first original image, and N, and i=0 is initialized.

[0164] In operation S702, the value of i is controlled to be incremented by 1.

[0165] In operation S703, it is determined whether i is less than N. In the case where it is determined that i is less than N, operation S704 is performed. In the case where it is determined that i is greater than or equal to N, operation S708 is performed.

[0166] In operation S704, a second target resolution is determined based on the image resolution and the expansion magnification of the expanded image after the (i-1)th image expansion.

[0167] In operation S705 , a fourth original image and a second mask image are generated based on the extended image after the (i-1)th image expansion using the target aspect ratio and the second target resolution.

[0168] In operation S706 , the image description text and the fourth original image are used as semantic references, and image generation is performed based on the second mask image to obtain a second target mask image.

[0169] In operation S707, the fourth original image is spliced ​​with the second target mask image to obtain an extended image after the i-th image extension. After operation S707 is completed, the process returns to execute operation S702.

[0170] In operation S708, the extended image after the i-th image expansion is determined as a target extended image.

[0171] According to the embodiments of the present disclosure, the following Figure 8A to Figure 8C The image is expanded twice and Fig. 8A Taking the first original image shown as an example, the image expansion effect is described.

[0172] Fig. 8A A schematic diagram of a first original image according to another embodiment of the present disclosure is schematically shown.

[0173] like Fig. 8A As shown, the original resolution of the first original image may be 512×512, the first target resolution may be 2560×1280, and the target aspect ratio may be 2: 1. It can be determined that the ratio between the first target resolution and the original resolution is 2.5, and the resolution may be expanded by 1.5 times the resolution of the original image each time, that is, the expansion ratio may be 1.5.

[0174] According to an embodiment of the present disclosure, by performing visual language recognition on the first original image, labels such as people, coins, blackboards, and charts can be obtained, and image description text can be generated based on the above labels.

[0175] According to an embodiment of the present disclosure, the second target resolution during the first expansion can be determined to be 768×768 based on the original resolution and the expansion ratio, and the fourth original image can be obtained accordingly.

[0176] According to an embodiment of the present disclosure, based on the target aspect ratio, a second mask image with a resolution of 768×768 may be added to the fourth original image.

[0177] According to an embodiment of the present disclosure, the second mask image may be processed using a diffusion model based on the image description text and the fourth original image to obtain an image after a primary expansion.

[0178] Figure 8B A schematic diagram of an image after one expansion according to another embodiment of the present disclosure is schematically shown.

[0179] like Figure 8B As shown, the resolution of the image after one expansion can be 1536×768.

[0180] According to an embodiment of the present disclosure, based on the resolution and expansion ratio of the image after the first expansion, the second target resolution during the second expansion can be determined to be 2304×1152, and the fourth original image can be obtained accordingly.

[0181] According to an embodiment of the present disclosure, two second mask images with resolutions of 2304×128 and 256×1280 respectively may be added to the fourth original image based on the target aspect ratio.

[0182] According to an embodiment of the present disclosure, the two second mask images may be processed using a diffusion model based on the image description text and the fourth original image to obtain a target extended image.

[0183] Figure 8C A schematic diagram of a target extended image according to another embodiment of the present disclosure is schematically shown.

[0184] like Figure 8C As shown, the resolution of the target extended image may be 2560×1280.

[0185] According to an embodiment of the present disclosure, by performing image expansion in multiple times, the quality of the generated image can be effectively improved, the content of the target expanded image can be kept consistent with the style of the first original image, and the display effect of the target expanded image can be improved.

[0186] Fig. 9 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0187] like Fig. 9 As shown, the method 900 may include operations S910 to S960.

[0188] In operation S910, in response to a first task start control being triggered, a first original image is acquired from a return result of a first image input control included in a display interface.

[0189] In operation S920, visual language recognition is performed on the first original image to obtain image description text.

[0190] In operation S930, a target aspect ratio and a first target resolution are obtained based on the returned results of each of the plurality of first configuration controls included in the display interface.

[0191] In operation S940, a second original image and a first mask image are generated based on the first original image using the target aspect ratio and the first target resolution.

[0192] In operation S950, the image description text and the second original image are used as semantic references, and image generation is performed based on the first mask image to obtain a first target mask image.

[0193] In operation S960, the first target mask image is spliced ​​with the second original image to obtain a target extended image.

[0194] According to an embodiment of the present disclosure, the display interface may refer to a display interface of a client application process running in an electronic device on a display screen of the electronic device. The electronic device may be any device that supports human-computer interaction.

[0195] According to an embodiment of the present disclosure, the first task start control may be a control for indicating the start of the image generation task, and the first task start control may be represented as a key or a button on the display interface. The user may touch and click the corresponding key or button, or the user may control the mouse to click the corresponding key or button to trigger the first task start control. When the first task start control is triggered, the electronic device may generate an image expansion request so that the client application process in the electronic device responds to the image expansion request to expand the image.

[0196] According to an embodiment of the present disclosure, the first image input control can be represented as a larger blank box on the real interface, and the user can input the first original image based on the image expansion task into the blank box by dragging, loading, etc. The first image input control can use the first original image as return information, which can facilitate the process in the electronic device to obtain the first original image by accessing the first image input control.

[0197] According to an embodiment of the present disclosure, the plurality of first configuration controls may include an input box type configuration control, a selection item type configuration control, a slider type configuration control, etc. Based on the different types of configuration controls, the user may input information through different operations. The information received by the configuration control may be used as return information, which may facilitate a process in the electronic device to obtain corresponding information by accessing the configuration control.

[0198] In another embodiment of the present disclosure, a method for expanding the first original image to obtain a target expanded image can be implemented using the image processing method of the embodiment of the present disclosure, which will not be described in detail herein.

[0199] Reference below Figures 10A to 10C , combined with specific embodiments Fig. 9 The method shown is further explained.

[0200] According to an embodiment of the present disclosure, the plurality of first configuration controls may include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control. The horizontal resolution configuration control, the vertical resolution configuration control, the horizontal expansion ratio configuration control, and the vertical resolution configuration control may all be configuration controls of an input box type.

[0201] According to an embodiment of the present disclosure, obtaining a target aspect ratio and a first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface may include the following operations:

[0202] Determine the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control; and determine the target aspect ratio based on the first target resolution.

[0203] According to an embodiment of the present disclosure, the horizontal expansion ratio information may refer to the ratio of the horizontal size of the extended image obtained after horizontal expansion based on the first original image to the horizontal size of the first original image. Correspondingly, the vertical expansion ratio information may refer to the ratio of the vertical size of the extended image obtained after vertical expansion based on the first original image to the vertical size of the first original image. Taking the multi-screen expansion scenario as an example, the first original image can be matched with a single screen, the horizontal expansion ratio information can be expressed as the number of horizontal screens, and the vertical expansion ratio information can be expressed as the number of vertical screens. For example, if the original single screen needs to be expanded to 2 rows and 3 columns with a total of 4 screens, the horizontal expansion ratio information can be set to 2, and the vertical expansion ratio information can be set to 3.

[0204] According to an embodiment of the present disclosure, the horizontal resolution information and the vertical resolution information may be expressed as a target resolution for adjusting the first original image when horizontal and vertical expansion is not performed, that is, a target resolution when the horizontal expansion ratio information and the vertical expansion ratio information are both expressed as 1. In a multi-screen expansion scenario, the horizontal resolution information and the vertical resolution information may be expressed as the resolution of an image on a single screen.

[0205] According to an embodiment of the present disclosure, the lateral resolution in the first target resolution can be obtained by multiplying the lateral resolution represented by the lateral resolution information by the lateral expansion ratio represented by the lateral expansion ratio information. Correspondingly, the longitudinal resolution in the first target resolution can be obtained by multiplying the longitudinal resolution represented by the longitudinal resolution information by the longitudinal expansion ratio represented by the longitudinal expansion ratio information.

[0206] According to an embodiment of the present disclosure, the horizontal resolution in the first target resolution may be ratioed with the vertical resolution in the first target resolution to obtain a target aspect ratio.

[0207] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include an expansion direction configuration control.

[0208] According to an embodiment of the present disclosure, generating a second original image and a first mask image based on a first original image by using a target aspect ratio and a first target resolution may include the following operations:

[0209] Based on the expansion direction information returned by the expansion direction configuration control, at least one target edge is determined from the four edges of the first original image; based on the target aspect ratio and the first target resolution, image expansion is performed with at least one target edge of the first original image as a starting point to obtain a first extended image; and a second original image and a first mask image are determined from the first extended image, wherein the second original image is the first original image.

[0210] According to an embodiment of the present disclosure, the extension direction configuration control may be a configuration control of a selection item type. For example, the extension direction configuration control may include four selection items representing "up", "down", "left" and "right" respectively, and the user may select one or more selection items from the four selection items to determine at least one target edge from the four edges of the first original image. Specifically, the user may select the two selection items "up" and "down", and the target edge may include the upper boundary and the lower boundary of the first original image.

[0211] According to an embodiment of the present disclosure, multiple first configuration controls may further include a candidate image quantity control. The candidate image quantity control may be an input box type configuration control or a slider type configuration control. The user may enter a number in the input box of the candidate image quantity control, or slide the reference cursor to the left or right at the slider of the candidate image quantity control to determine the candidate image quantity information. The candidate image quantity information may be represented as the number of canvas images generated before the image is expanded and generated. Specifically, depending on the expansion direction of the selected image, when the expansion direction includes two relative directions, such as including the "up" and "down" directions at the same time, or including the "left" and "right" directions at the same time, the pixels of the canvas expanded in the two directions may be different, thereby generating multiple selectable canvas images. The candidate image quantity information is represented as the number of generated canvas images.

[0212] According to an embodiment of the present disclosure, based on the target aspect ratio and the first target resolution, image expansion is performed with at least one target edge of the first original image as a starting point to obtain a first extended image, which may include the following operations:

[0213] Based on the candidate image quantity information returned by the candidate image quantity control, determine the number of candidate images; based on the target aspect ratio and the first target resolution, perform image expansion with at least one target edge of the first original image as a starting point to obtain a fifth extended image of the number of candidate images; send the fifth extended image of the number of candidate images to the first image output control, wherein the first image output control is configured to display the fifth extended image of the number of candidate images on a display interface; and in response to a selection request for the fifth extended image of the number of candidate images, determine a first extended image from the fifth extended image of the number of candidate images.

[0214] According to an embodiment of the present disclosure, the fifth extended image is the canvas image obtained by expansion, and the fifth extended image may include the original first original image and extended partial images with different shapes, colors, etc. obtained by expansion.

[0215] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a generation speed configuration control. The generation speed configuration control may be an input box type configuration control or a slide bar type configuration control. The user may enter a number in the input box of the generation speed configuration control, or slide the reference cursor to the left or right at the slide bar of the generation speed configuration control to determine the generation speed information.

[0216] According to an embodiment of the present disclosure, the generation speed information can be represented as the number of rounds of image expansion, that is, the number of image expansions. The smaller the value represented by the generation speed information, the slower the image generation speed is, and accordingly, the more image expansions are determined by the generation speed information.

[0217] According to an embodiment of the present disclosure, a plurality of first configuration controls may further include a single maximum expansion pixel control, which may be an input box type configuration control or a slider type configuration control. A user may enter a number in the input box of the single maximum expansion pixel control, or slide the reference cursor to the left or right at the slider of the single maximum expansion pixel control to determine the single maximum expansion pixel information. The single maximum expansion pixel information may be expressed as the maximum allowable horizontal pixel expansion amount and vertical pixel expansion amount for one image expansion. For example, if the single maximum expansion pixel information is expressed as 256, it means that when a single image is expanded, the horizontal pixels are expanded or reduced by a maximum of 256 pixels, and the vertical pixels are expanded or reduced by a maximum of 256 pixels. When the difference between the first target pixel and the original pixel of the first original image is greater than 256 pixels, the image expansion may be performed multiple times, that is, the number of image expansions is determined to be greater than 1.

[0218] According to an embodiment of the present disclosure, the number of image expansions can be determined based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image and the first target resolution; and when the number of image expansions is greater than 1, based on the number of image expansions, multiple image expansions are performed with the first original image and the image description text as semantic references to obtain a target expanded image.

[0219] According to an embodiment of the present disclosure, specifically, the first image expansion number can be determined based on the generation speed information returned by the generation speed configuration control; the second image expansion number can be determined based on the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image and the first target resolution; and the maximum value of the first image expansion number and the second image expansion number is taken as the determined image expansion number.

[0220] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include an image style configuration control. The image style configuration control may be a configuration control of a selection item type, and the configuration control may include a plurality of selection items representing image styles, for example, a selection item representing an image style of "poster", a selection item representing an image style of "landscape", a selection item representing an image style of "architecture", etc. The user may determine the image style information by selecting the selection item.

[0221] According to an embodiment of the present disclosure, performing visual language recognition on the first original image to obtain image description text may include the following operations:

[0222] Based on the image style information returned by the image style configuration control, an image extension keyword is obtained; the first original image is processed using a visual language pre-trained model to obtain at least one object text; and based on the at least one object text and the image extension keyword, an image description text is obtained.

[0223] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include a creativity configuration control. The creativity configuration control may be an input box type configuration control or a slide bar type configuration control. The user may enter a number in the input box of the creativity configuration control, or slide the reference cursor to the left or right at the slide bar of the creativity configuration control to determine the creativity information.

[0224] According to an embodiment of the present disclosure, the creativity information may be represented as the similarity between the generated target extended image and the original first original image. The higher the creativity represented by the creativity information, the lower the similarity between the generated target extended image and the original first original image.

[0225] According to an embodiment of the present disclosure, the influence of the image coding features and the text coding features on the generation process of the target extended image can be changed by weighting the image coding features and the text coding features, thereby achieving control of the similarity between the generated target extended image and the original first original image. The higher the creativity represented by the creativity information, the smaller the weight of the determined image coding features and the text coding features can be.

[0226] According to an embodiment of the present disclosure, specifically, taking the image description text and the second original image as semantic references, performing image generation based on the first mask image to obtain the first target mask image may include the following operations:

[0227] The second original image and the image description text are respectively encoded by a multimodal encoder to obtain initial image coding features and initial text coding features; based on the creativity information returned by the creativity configuration control, the initial image coding features and the initial text coding features are corrected to obtain image coding features and text coding features; the first mask image is initialized by Gaussian noise to obtain a noisy image; the image coding features and the text coding features are used as semantic benchmarks, and the noise image is processed by an image information creator to obtain target output features; and the target output features are decoded by a model decoder to obtain a first target mask image.

[0228] According to an embodiment of the present disclosure, the current weight of each of the initial image coding feature and the initial text coding feature may be 1. A weight with a value of 0 to 1 may be determined based on the creativity information, and the initial image coding feature and the initial text coding feature may be corrected using the weight. Specifically, the weight may be vector multiplied with the initial image coding feature and the initial text coding feature to obtain the image coding feature and the text coding feature.

[0229] Fig. 10A A schematic diagram of a first display interface according to an embodiment of the present disclosure is schematically shown.

[0230] like Fig. 10AAs shown, the first display interface can be represented as a portion of the display interface before image expansion. The display interface can include a first task start control, a first image input control, and configuration controls such as a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, a vertical resolution configuration control, an expansion direction configuration control, a candidate image quantity control, a generation speed configuration control, a single maximum expansion pixel control, an image style configuration control, and a creativity configuration control. Before image expansion, the user can place the first original image into the first image input control and complete information configuration in each configuration control. Specifically, the configuration information filled in by the user may include setting the generation speed information to 2, the maximum single expansion pixel to 256, the image style information to "poster", the creativity information to 2, the expansion direction information to "left" and "right", that is, expanding to the left and right sides based on the first original image, the number of candidate images information is set to 4, that is, 4 candidate fifth extended images will be generated for the user to choose, the horizontal resolution information is set to 1024, the vertical resolution information is set to 768, the horizontal expansion ratio information is set to 1, and the vertical expansion ratio information is set to 1, which can determine that the first target resolution is 1024×768, and the target aspect ratio is 4:3.

[0231] According to an embodiment of the present disclosure, after determining that the first original image has been input and the configuration has been completed, the user can start executing the image expansion task by clicking the first task start control. Alternatively, after determining that the first original image has been input but the configuration has not been completed, if the user clicks the first task start control, the corresponding configuration controls can be filled in using the preset values ​​or initial values ​​corresponding to the respective unconfigured configuration controls, and then the image expansion task can be started.

[0232] According to an embodiment of the present disclosure, the first display interface may include other functional controls in addition to the above-mentioned controls. For example, the first display interface may also include a resolution display control, and the resolution display control may be used to display the original resolution of the first original image. Specifically, after the user inputs the first original image into the first image input control, an image input request may be generated. In response to the image input request related to the first original image, the resolution information of the first original image may be determined; and the resolution information may be sent to the resolution display control, wherein the resolution display control is configured to display the original resolution of the first original image on the display interface based on the resolution information. Fig. 10A As shown, after the first original image is input into the first image input control, the original resolution of the first original image can be displayed as 640×640.

[0233] According to an embodiment of the present disclosure, for another example, the first display interface may further include an initialization control, and the initialization control may be used to initialize configuration information of the plurality of first configuration controls.

[0234] According to an embodiment of the present disclosure, after it is determined that the first task start control is clicked, a fifth extended image of the number of candidate images may be displayed on the display interface of the electronic device.

[0235] Fig. 10B A schematic diagram of a second display interface according to an embodiment of the present disclosure is schematically shown.

[0236] like Fig. 10B As shown, the second display interface may represent the display interface portion during the execution of the image expansion task, and the second display interface may be displayed side by side with the first display interface. The second display interface may include a first image output control, and the first image output control may include a selection box, in which the fifth extended image may be displayed. Optionally, the selection box may also include thumbnails of each fifth extended image, and the user may select the thumbnail to determine the fifth extended image displayed in the selection box.

[0237] According to an embodiment of the present disclosure, the first image output control may further include a plurality of selection items, wherein the selection items of the number of first candidate images in the plurality of selection items may correspond one to one with the fifth extended image of the number of candidate images. The first image output control may further include a display box, and the user may determine the final selected first extended image by checking the plurality of selection items. The first extended image displayed in the display box may be different from the fifth extended image displayed in the selection box.

[0238] According to an embodiment of the present disclosure, the second display interface may optionally further include a key or button-type control, which may indicate that the first extended image has been selected and the generation of the target extended image has begun. The name of the control in the second display interface may be, for example, Fig. 10B The user can click the “Start Super Resolution” control to start generating the target extended image.

[0239] Fig. 10C A schematic diagram of a third display interface according to an embodiment of the present disclosure is schematically shown.

[0240] like Fig. 10C As shown, the third display interface may represent the display interface portion after the image expansion task is completed, and the third display interface may be displayed side by side with the first display interface. The third display interface may include a second image output control.

[0241] According to an embodiment of the present disclosure, the target expanded image may be sent to the second image output control, wherein the second image output control is configured to display the target expanded image on the display interface.

[0242] Fig.11 A block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure.

[0243] like Fig.11 As shown, the image processing device 1100 includes a first recognition module 1110 , a first generation module 1120 , a first image generation module 1130 and a first stitching module 1140 .

[0244] The first recognition module 1110 is used to perform visual language recognition on the first original image in response to the image expansion request to obtain image description text.

[0245] The first generating module 1120 is configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution included in the image extension request.

[0246] The first image generation module 1130 is configured to generate an image based on the first mask image by taking the image description text and the second original image as semantic references to obtain a first target mask image.

[0247] The first stitching module 1140 is used to stitch the first target mask image with the second original image to obtain a target extended image.

[0248] According to an embodiment of the present disclosure, the first generating module 1120 includes a first generating submodule and a second generating submodule.

[0249] The first generating submodule is used to perform image expansion based on a target aspect ratio and a first target resolution and taking at least one edge of the first original image as a starting point to obtain a first expanded image.

[0250] The second generating submodule is used to determine a second original image and a first mask image from the first extended image, wherein the second original image is the first original image.

[0251] According to an embodiment of the present disclosure, the first generation module 1120 includes a third generation submodule, a fourth generation submodule, and a fifth generation submodule.

[0252] The third generating submodule is used to perform scaling processing on the first original image based on the original resolution and the preset resolution of the first original image to obtain a third original image.

[0253] The fourth generating submodule is used to perform image expansion based on the target aspect ratio and taking at least one side of the third original image as a starting point to obtain a second expanded image.

[0254] The fifth generating submodule is used to determine the second original image and the first mask image from the second extended image when the first target resolution is equal to the preset resolution.

[0255] According to an embodiment of the present disclosure, the first generation module 1120 further includes a sixth generation submodule and a seventh generation submodule.

[0256] The sixth generating submodule is used to scale the first extended image based on the first target resolution and the preset resolution to obtain a third extended image when the first target resolution is not equal to the preset resolution.

[0257] The seventh generating submodule is used to determine the second original image and the first mask image from the third extended image.

[0258] According to an embodiment of the present disclosure, the image processing apparatus 1100 further includes a first expansion module.

[0259] The first expansion module is used to perform N image expansions based on the first target resolution and the original resolution of the first original image and the image description text as semantic references to obtain a target expanded image, where N is a positive integer.

[0260] According to an embodiment of the present disclosure, the preset condition includes any one of the following: a ratio between the first target resolution and the original resolution of the first original image is less than a first threshold value; a ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold value; and the number of image extensions included in the image extension request is greater than 1.

[0261] According to an embodiment of the present disclosure, the first extension module includes a first extension submodule, a second extension submodule, a third extension submodule, a fourth extension submodule and a fifth extension submodule.

[0262] The first expansion submodule is used to determine an expansion ratio based on the first target resolution, the original resolution of the first original image and N.

[0263] The second expansion submodule is used to determine the second target resolution based on the image resolution and expansion ratio of the extended image after the i-1th image expansion when the first target resolution and the image resolution of the extended image after the i-1th image expansion are not equal, wherein i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution.

[0264] The third expansion submodule is used to generate a fourth original image and a second mask image based on the expanded image after the (i-1)th image expansion by using the target aspect ratio and the second target resolution.

[0265] The fourth expansion submodule is used to use the image description text and the fourth original image as semantic references, perform image generation based on the second mask image, and obtain a second target mask image.

[0266] The fifth expansion submodule is used to perform image splicing on the fourth original image and the second target mask image to obtain an expanded image after the i-th image expansion.

[0267] According to an embodiment of the present disclosure, the first expansion module further includes a sixth expansion submodule.

[0268] The sixth expansion submodule is used to determine the extended image after the i-th image expansion as the target extended image when the first target resolution is equal to the image resolution of the extended image after the i-th image expansion.

[0269] According to an embodiment of the present disclosure, the first image generating module 1130 includes a first image generating submodule, a second image generating submodule, a third image generating submodule and a fourth image generating submodule.

[0270] The first image generation submodule is used to use a multimodal encoder to encode the second original image and the image description text respectively to obtain image encoding features and text encoding features.

[0271] The second image generation submodule is used to initialize the first mask image using Gaussian noise to obtain a noise image.

[0272] The third image generation submodule is used to use the image coding features and the text coding features as semantic benchmarks, use the image information creator to process the noisy image, and obtain the target output features.

[0273] The fourth image generation submodule is used to use the model decoder to decode the target output features to obtain a first target mask image.

[0274] According to an embodiment of the present disclosure, an image information creator includes an encoding module, a decoding module and an intermediate module.

[0275] According to an embodiment of the present disclosure, the third image generating submodule includes a first image generating unit, a second image generating unit and a third image generating unit.

[0276] The first image generating unit is used to process the image coding features, the text coding features and the noise image by using the coding module to obtain the first noise image features outputted by each of the M coding function blocks included in the coding module, wherein M is a positive integer.

[0277] The second image generating unit is used to process the first noise image feature output by the Mth encoding function block by using the intermediate module to obtain the second noise image feature.

[0278] The third image generating unit is used to use the decoding module to process the second noise image feature, the first noise image feature outputted by each of the M encoding function blocks, the image encoding feature and the text encoding feature to obtain the target output feature.

[0279] According to an embodiment of the present disclosure, the M encoding functional blocks are connected through cross attention blocks.

[0280] According to an embodiment of the present disclosure, the first image generating unit includes a first image generating subunit and a second image generating subunit.

[0281] The first image generation subunit is used to use the jth coding function block to process the input features of the jth coding function block to obtain the first noise image features output by the jth coding function block, wherein j is a positive integer less than or equal to M, and the input features of the first coding function block are represented as a noise image.

[0282] The second image generation subunit is used to process the first noise image features, image coding features, and text coding features output by the jth coding functional block using the cross-attention block between the jth coding functional block and the j+1th coding functional block when j is less than M, so as to obtain the input features of the j+1th coding functional block.

[0283] According to an embodiment of the present disclosure, the decoding module includes M decoding functional blocks, and the M decoding functional blocks are connected by cross attention blocks.

[0284] According to an embodiment of the present disclosure, the third image generating unit includes a third image generating sub-unit, a fourth image generating sub-unit and a fifth image generating sub-unit.

[0285] The third image generation subunit is used to use the kth decoding function block to process the input features of the kth decoding function block and the first noise image features output by the kth encoding function block to obtain the decoding features output by the kth decoding function block, wherein k is a positive integer less than or equal to M, and the input features of the first decoding function block are represented as the second noise image features.

[0286] The fourth image generation subunit is used to process the decoding features, image coding features and text coding features output by the kth decoding functional block using the cross-attention block between the kth decoding functional block and the k+1th decoding functional block when k is less than M, so as to obtain the input features of the k+1th decoding functional block.

[0287] The fifth image generation subunit is used to determine, when k is equal to M, that the target output feature is a decoding feature output by the kth decoding functional block.

[0288] According to an embodiment of the present disclosure, the intermediate module, the encoding functional block and the decoding functional block are gated convolution blocks.

[0289] According to an embodiment of the present disclosure, the second image generating unit includes a sixth image generating subunit, a seventh image generating subunit, an eighth image generating subunit, a ninth image generating subunit, and a tenth image generating subunit.

[0290] The sixth image generation subunit is used to perform layer normalization processing on the first noise image feature output by the Mth encoding function block to obtain a first sub-feature.

[0291] The seventh image generation subunit is used to process the first sub-feature by using a gated convolutional layer to obtain a second sub-feature.

[0292] The eighth image generating subunit is used to perform layer normalization processing on the second sub-feature to obtain a third sub-feature.

[0293] The ninth image generation subunit is used to process the third sub-feature using a perceptron to obtain a fourth sub-feature.

[0294] The tenth image generating subunit is used to obtain the second noise image feature based on the first noise image feature and the fourth sub-feature output by the Mth encoding functional block.

[0295] According to an embodiment of the present disclosure, the gated convolution layer includes P gated image generation units, where P is a positive integer.

[0296] According to an embodiment of the present disclosure, the seventh image generating subunit includes a first image generating component, a second image generating component, a third image generating component and a fourth image generating component.

[0297] The first image generating component is used to perform convolution processing on the first sub-feature to obtain a fifth sub-feature.

[0298] The second image generating component is used to perform deep convolution processing on the fifth sub-feature to obtain a deep convolution feature.

[0299] The third image generation component is used to perform dot multiplication processing on the deep convolution feature and the gated output feature output by the p-1th gated image generation unit for the pth gated image generation unit to obtain the sixth sub-feature of the pth gated image generation unit, where p is a positive integer less than or equal to P.

[0300] The fourth image generation component is used to perform convolution processing on the sixth sub-feature of the p-th gated image generation unit to obtain a gated output feature output by the p-th gated image generation unit, wherein the gated output feature of the 0th gated image generation unit is represented as the fifth sub-feature, and the gated output feature of the P-th gated image generation unit is represented as the second sub-feature.

[0301] According to an embodiment of the present disclosure, the first identification module 1110 includes a first identification submodule and a second identification submodule.

[0302] The first recognition submodule is used to process the first original image using the visual language pre-training model to obtain at least one object text.

[0303] The second recognition submodule is used to obtain an image description text based on at least one object text.

[0304] According to an embodiment of the present disclosure, the image extension request further includes an image extension keyword.

[0305] According to an embodiment of the present disclosure, the first identification module 1110 includes a third identification submodule.

[0306] The third recognition submodule is used to obtain an image description text based on at least one object text and an image extension keyword.

[0307] It should be noted that the image processing device part in the embodiment of the present disclosure corresponds to the image processing method part in the embodiment of the present disclosure. The description of the image processing device part specifically refers to the image processing method part, which will not be repeated here.

[0308] Fig.12 A block diagram of an image processing device according to another embodiment of the present disclosure is schematically shown.

[0309] like Fig.12 As shown, the image processing device 1200 includes a first acquisition module 1210 , a second recognition module 1220 , a second acquisition module 1230 , a second generation module 1240 , a second image generation module 1250 and a second stitching module 1260 .

[0310] The first acquisition module 1210 is configured to acquire a first original image from a return result of a first image input control included in the display interface in response to the first task start control being triggered.

[0311] The second recognition module 1220 is used to perform visual language recognition on the first original image to obtain image description text.

[0312] The second acquisition module 1230 is used to obtain a target aspect ratio and a first target resolution based on the return results of each of the multiple first configuration controls included in the display interface.

[0313] The second generating module 1240 is configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution.

[0314] The second image generation module 1250 is used to generate an image based on the first mask image by taking the image description text and the second original image as semantic references to obtain a first target mask image.

[0315] The second stitching module 1260 is used to stitch the first target mask image with the second original image to obtain a target extended image.

[0316] According to an embodiment of the present disclosure, the plurality of first configuration controls include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control.

[0317] According to an embodiment of the present disclosure, the second acquisition module 1230 includes a first acquisition submodule and a second acquisition submodule.

[0318] The first acquisition submodule is used to determine the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control.

[0319] The second acquisition submodule is used to determine a target aspect ratio based on the first target resolution.

[0320] According to an embodiment of the present disclosure, the plurality of first configuration controls further include an expansion direction configuration control.

[0321] According to an embodiment of the present disclosure, the second generation module 1240 includes an eighth generation submodule, a ninth generation submodule, and a tenth generation submodule.

[0322] The eighth generating submodule is used to determine at least one target edge from the four edges of the first original image based on the extension direction information returned by the extension direction configuration control.

[0323] The ninth generating submodule is used to perform image expansion based on the target aspect ratio and the first target resolution and taking at least one target edge of the first original image as a starting point to obtain a first expanded image.

[0324] The tenth generating submodule is used to determine a second original image and a first mask image from the first extended image, wherein the second original image is the first original image.

[0325] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a candidate image quantity control.

[0326] According to an embodiment of the present disclosure, the ninth generation submodule includes a first generation unit, a second generation unit, a third generation unit and a fourth generation unit.

[0327] The first generating unit is used to determine the number of candidate images based on the candidate image number information returned by the candidate image number control.

[0328] The second generating unit is used to perform image expansion based on the target aspect ratio and the first target resolution, taking at least one target edge of the first original image as a starting point, to obtain fifth expanded images of the same number as the candidate images.

[0329] The third generating unit is used to send a fifth extended image of the number of candidate images to the first image output control, wherein the first image output control is configured to display the fifth extended image of the number of candidate images on the display interface.

[0330] The fourth generating unit is configured to determine a first extended image from the fifth extended images of the number of candidate images in response to a selection request for the fifth extended image of the number of candidate images.

[0331] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a generation speed configuration control and a single maximum expansion pixel control.

[0332] According to an embodiment of the present disclosure, the image processing apparatus 1200 further includes a first determining module and a second expanding module.

[0333] The first determination module is used to determine the number of image expansion times based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image and the first target resolution.

[0334] The second expansion module is used to perform multiple image expansions based on the number of image expansions when the number of image expansions is greater than 1, using the first original image and the image description text as semantic references to obtain a target expanded image.

[0335] According to an embodiment of the present disclosure, the plurality of first configuration controls further include an image style configuration control.

[0336] According to an embodiment of the present disclosure, the second identification module 1220 includes a fourth identification submodule, a fifth identification submodule and a sixth identification submodule.

[0337] The fourth identification submodule is used to obtain image extension keywords based on the image style information returned by the image style configuration control.

[0338] The fifth recognition submodule is used to process the first original image using the visual language pre-training model to obtain at least one object text.

[0339] The fifth recognition submodule is used to obtain image description text based on at least one object text and image extension keywords.

[0340] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a creativity configuration control.

[0341] According to an embodiment of the present disclosure, the second image generating module 1250 includes a fifth image generating submodule, a sixth image generating submodule, a seventh image generating submodule, an eighth image generating submodule and a ninth image generating submodule.

[0342] The fifth image generation submodule is used to use a multimodal encoder to encode the second original image and the image description text respectively to obtain initial image encoding features and initial text encoding features.

[0343] The sixth image generation submodule is used to modify the initial image coding features and the initial text coding features based on the creativity information returned by the creativity configuration control to obtain image coding features and text coding features.

[0344] The seventh image generation submodule is used to initialize the first mask image using Gaussian noise to obtain a noise image.

[0345] The eighth image generation submodule is used to use the image coding features and the text coding features as semantic benchmarks, use the image information creator to process the noisy image, and obtain the target output features.

[0346] The ninth image generation submodule is used to decode the target output features using the model decoder to obtain a first target mask image.

[0347] According to an embodiment of the present disclosure, the image processing device 1200 further includes a second determining module and a first sending module.

[0348] The second determining module is used to determine the resolution information of the first original image in response to an image input request related to the first original image.

[0349] The first sending module is used to send resolution information to the resolution display control, wherein the resolution display control is configured to display the original resolution of the first original image on the display interface based on the resolution information.

[0350] According to an embodiment of the present disclosure, the image processing apparatus 1200 further includes a second sending module.

[0351] The second sending module is used to send the target extended image to the second image output control, wherein the second image output control is configured to display the target extended image on the display interface.

[0352] It should be noted that the image processing device part in another embodiment of the present disclosure corresponds to the image processing method part in another embodiment of the present disclosure. The description of the image processing device part specifically refers to the image processing method part, which will not be repeated here.

[0353] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0354] For example, any multiple of the first recognition module 1110, the first generation module 1120, the first image generation module 1130 and the first splicing module 1140, or the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250 and the second splicing module 1260 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first recognition module 1110, the first generation module 1120, the first image generation module 1130 and the first stitching module 1140, or the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250 and the second stitching module 1260 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, the first recognition module 1110, the first generation module 1120, the first image generation module 1130 and the first stitching module 1140, or at least one of the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250 and the second stitching module 1260 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0355] According to the embodiments of the present disclosure, the image processing method or image processing device as described above can also be used in various image generation application scenarios. For example, the image processing method can be applied in dynamic image generation scenarios, including but not limited to using the image processing method to generate template images of dynamic images, and using the image processing method to fine-tune images in the generation process of dynamic image frames of different frames, etc., which are not limited here.

[0356] Fig.13 The flowchart of the dynamic image generation method according to the embodiment of the present disclosure is schematically shown.

[0357] like Fig.13As shown, the method 1300 includes operations S1310 to S1330.

[0358] In operation S1310, in response to the second task starting control being triggered, a prompt word text is obtained based on a return result of a prompt word input control included in the display interface.

[0359] In operation S1320, the prompt word text is used as a semantic reference, and image generation is performed based on the initial template image to obtain a first template image.

[0360] In operation S1330, a target dynamic image is generated based on the first template image according to a return result of a dynamic image configuration control included in the display interface.

[0361] According to an embodiment of the present disclosure, the second task start control may be a control for indicating the start of the dynamic image generation task, and the second task start control may be represented as a key or a button on the display interface. The user may touch and click the corresponding key or button, or the user may control the mouse to click the corresponding key or button to trigger the second task start control. When the second task start control is triggered, the electronic device may generate a template image generation request so that the client application process in the electronic device responds to the template image generation request to generate the first template image.

[0362] According to an embodiment of the present disclosure, the prompt word input control can be a control of the input box type. The user can enter one or more prompt words, or enter a piece of text in the input box of the prompt word input control. The prompt word input control can use the one or more prompt words or a piece of text as return information. The electronic device can obtain the one or more prompt words or a piece of text by accessing the prompt word input control, and process the one or more prompt words or a piece of text to obtain the prompt word text. For example, when generating an image for product promotion, the user can enter prompt words or text about the product type, product location, background environment description, etc. in the input box of the prompt word input control, such as a round bottle on a stone, cloudy, grassland, trees, flowers, and a mountain peak faintly appearing in the distance.

[0363] According to an embodiment of the present disclosure, the second task starting control and the prompt word input control may be located in the same page of the display interface.

[0364] According to an embodiment of the present disclosure, the initial template image may be a blank image with a set resolution and aspect ratio.

[0365] According to an embodiment of the present disclosure, the method of generating a first template image based on a prompt word text can be implemented using the image processing method of an embodiment of the present disclosure, which will not be described in detail here.

[0366] According to an embodiment of the present disclosure, the first template image can be fine-tuned to generate multiple dynamic image frames. There can be small differences between the dynamic image frames of adjacent frames generated by fine-tuning, and the multiple dynamic image frames can be played in sequence to achieve an animation effect. Therefore, the multiple dynamic image frames can be combined to obtain the target dynamic image to be generated.

[0367] Reference below Figure 14 to Figure 16 , Figures 17A to 17B , Figures 18A to 18B , Fig.19 and Fig. 20 , combined with specific embodiments Fig.13 The method shown is further explained.

[0368] Fig.14 A schematic diagram of a fourth display interface according to an embodiment of the present disclosure is schematically shown.

[0369] like Fig.14 As shown, the fourth display interface can be represented as a display interface portion before starting the dynamic image generation task. The fourth display interface may include a prompt word input control, a second task start control, and an image display control. The prompt word input control can be represented as an input box in the fourth display interface, and the user can enter the image description text in the input box to complete the prompt word design. The second task start control can be represented as two buttons on the fourth display interface, including a "start generation" button for indicating the start of the task and a "restore settings" button for indicating the initialization task. When performing a dynamic image generation task, if this task is the first task after the start, the user can directly enter the image description text in the input box, and then click the "start generation" button to start the generation of the first template image, and display the generated first template image in the image display control. If this task is the second or more than the second task, the user can first click the "restore settings" button to initialize the text content in the input box and the image content in the image display control, respectively, and then enter the image description text and generate the first template image.

[0370] According to an embodiment of the present disclosure, the image description text input by the user in the input box can be used as the return result of the prompt word input control. The electronic device can obtain the return result of the prompt word input control, and split the image description text in the return result to obtain multiple prompt words. The multiple prompt words can each have their own type, and accordingly, the template text can include multiple blank fields, each of which can have its own type. According to the type of each prompt word, the multiple prompt words can be written into the template text to obtain the prompt word text.

[0371] According to an embodiment of the present disclosure, the template text may be pre-configured by a developer. Alternatively, the electronic device may pre-store multiple template texts, and the electronic device may select a suitable template text from the multiple template texts based on the semantic information of each of the multiple prompt words, or select the template text based on the return results of other controls.

[0372] According to an embodiment of the present disclosure, optionally, the fourth display interface may further include an image style configuration control. The image style configuration control may be a configuration control of a selection item type, and the image style configuration control may include multiple selection items representing image styles, such as a selection item representing that the image style is "landscape", a selection item representing that the image style is "starry sky", a selection item representing that the image style is "beach", etc. The user may determine the image style information by selecting the selection item. The image style information may be used to determine a suitable template text from a variety of template texts.

[0373] According to an embodiment of the present disclosure, optionally, the fourth display interface may further include an aspect ratio configuration control. The aspect ratio configuration control may be a configuration control of a selection item type, and the aspect ratio configuration control may include multiple selection items representing the aspect ratio of the generated image, for example, including a selection item representing that the aspect ratio of the generated image is "1:1", a selection item representing that the aspect ratio of the generated image is "4:3", a selection item representing that the aspect ratio of the generated image is "16:9", a selection item representing that the aspect ratio of the generated image is "3:4", a selection item representing that the aspect ratio of the generated image is "9:16", etc. The user may determine the aspect ratio of the generated first template image by selecting the above selection items.

[0374] According to an embodiment of the present disclosure, the fourth display interface may further include a first window for displaying the generated first template image.

[0375] According to an embodiment of the present disclosure, optionally, the return result of the prompt word input control can also be directly used as the prompt word text. In this case, the user needs to enter the complete prompt word text in the input box of the prompt word input control. For example, the user can enter "cloudy, grassland, trees, manifestation, mountains looming in the distance, and a beautiful small jar on the stone", and after clicking the "start generating" button, the first template image generated based on the user input text can be displayed in the first window.

[0376] According to an embodiment of the present disclosure, the fourth display interface may also include at least one interface jump control, which can be used to jump from the current fourth display interface to other display interfaces to complete subsequent operations on the first template image in other display interfaces.

[0377] According to an embodiment of the present disclosure, the at least one interface jump control may include an interface jump control "dynamic image generation" for jumping to a fifth display interface, and the fifth display interface may be a display interface when a user performs a dynamic image generation task. After the user clicks the interface jump control, the electronic device may send the generated first template image to the control in the fifth display interface, so as to generate an image using the control in the fifth display interface.

[0378] Fig.15 A schematic diagram of a fifth display interface according to an embodiment of the present disclosure is schematically shown.

[0379] like Fig.15 As shown, similar to the fourth display interface, the fifth display interface may also include a task start control, and the task start control may include a "start generating" button for indicating starting a task and a "restore setting" button for indicating initializing a task.

[0380] According to an embodiment of the present disclosure, the fifth display interface may further include a dynamic image configuration control, which may be a configuration control of a selection item type, and the dynamic image configuration control may include multiple selection items indicating the animation direction of the dynamic image to be generated, for example, including a selection item indicating that the animation direction of the dynamic image to be generated is a "left-right" direction, a selection item indicating that the animation direction of the dynamic image to be generated is a "up-down" direction, a selection item indicating that the animation direction of the dynamic image to be generated is a "circular" direction, i.e., a selection item for a look-around direction, etc. The user may determine the animation effect of the target dynamic image to be generated by selecting the above selection items.

[0381] According to an embodiment of the present disclosure, the fifth display interface may also include a second window for displaying the image based on which the dynamic image is generated, that is, the first template image, and a third window for playing the generated target dynamic image. After the user clicks on the interface jump control for jumping to the fifth display interface in the fourth display interface, the first template image generated in the fourth display interface can be transmitted to the fifth display interface and displayed in the second window. The user can select the desired animation direction from the multiple options of the dynamic image configuration control of the fifth display interface, and the information of the selected animation direction can be returned to the electronic device as the return result of the dynamic image configuration control. After the user clicks the "Start Generation" button, the electronic device can start generating the target dynamic image based on the first template image.

[0382] According to an embodiment of the present disclosure, the electronic device may generate a target dynamic image based on the first template image, including the following operations:

[0383] Performing depth prediction on the first template image to obtain a depth feature map of the first template image; splitting the first template image into a plurality of first connected domain images based on the depth feature map; generating a plurality of dynamic image frames based on the plurality of first connected domain images according to the return result of the dynamic image configuration control; and combining the plurality of dynamic image frames to obtain a target dynamic image.

[0384] According to an embodiment of the present disclosure, depth prediction of the first template image can be implemented using various depth prediction algorithms or models, which are not limited here. The predicted depth feature map can be used to represent the depth information of each pixel in the first template image.

[0385] According to the embodiments of the present disclosure, based on the depth information of each pixel in the image, the pixels with similar depths expressed by the depth information and adjacent positions can be classified into the same connected domain. That is, based on the depth information expressed in the depth feature map, the multiple pixels contained in the first template image can be classified into multiple connected domains, and the pixels contained in each connected domain can be extracted to form a connected domain image.

[0386] According to an embodiment of the present disclosure, the return result of the dynamic image configuration control can be expressed as the animation direction information of the target dynamic image to be generated. According to the return result of the dynamic image configuration control, the processing strategy of each of the multiple first connected domain images associated with each dynamic image frame can be determined. The processing strategies of each of the multiple first connected domain images associated with each dynamic image frame can be different.

[0387] According to an embodiment of the present disclosure, the plurality of first connected domain images related to each dynamic image frame may be processed respectively using respective processing strategies of the plurality of first connected domain images to obtain each dynamic image frame, so as to generate multiple dynamic image frames.

[0388] According to the embodiments of the present disclosure, specifically, based on the processing strategy of each first connected domain image related to each dynamic image frame, each first connected domain image can be translated, filled and / or rotated to obtain a fifth mask image corresponding to each first connected domain image; each first connected domain image is used as a semantic reference, and image generation is performed based on the fifth mask image corresponding to each first connected domain image to obtain multiple second connected domain images related to each dynamic image frame; and the multiple second connected domain images related to each dynamic image frame are spliced ​​to obtain each dynamic image frame.

[0389] For example, the multiple first connected domain images may include a first connected domain image representing the foreground part and a first connected domain image representing the background part. When the animation direction information indicates the "left" direction, the first connected domain image representing the foreground part may not be processed when generating each frame of the dynamic image; for each first connected domain image representing the background part, the first connected domain image representing a shallower depth may be translated to the right by fewer pixels, and the first connected domain image representing a deeper depth may be translated to the right by more pixels. The specific method of processing the multiple first connected domain images to obtain the multiple second connected domain images is not limited here.

[0390] According to an embodiment of the present disclosure, optionally, after obtaining a plurality of second connected domain images, the plurality of second connected domains can be used as semantic references to generate each dynamic image frame based on the first template image, so as to repair overlapping areas or blank areas generated by splicing the plurality of second connected domain images.

[0391] According to an embodiment of the present disclosure, as an optional implementation, various processing operations may be performed on the first template image to obtain a new template image, and the new template image may be used to generate a target dynamic image. The various processing operations may include, but are not limited to, element replacement, image expansion, partial redrawing, etc. of the first template image, which are not limited here.

[0392] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as element replacement as an example, the user may use an existing reference image to replace the object element in the generated first template image to generate a new template image.

[0393] Fig.16 A schematic diagram of a fourth display interface according to another embodiment of the present disclosure is schematically shown.

[0394] like Fig.16 As shown, the fourth display interface may also include an image input control, which may provide an image input interface, which may be represented as a window on the fourth display interface. The user may click on the window to jump to the file selection interface to select and upload a reference image, or the user may drag the reference image to the window to upload the reference image. The electronic device may obtain a reference image based on the return result of the second image input control included in the display interface; and generate a second template image based on the first template image and the reference image. Afterwards, the target dynamic image may be generated based on the second template image according to the return result of the dynamic image configuration control included in the display interface.

[0395] According to an embodiment of the present disclosure, generating a second template image based on the first template image and the reference image may be to use the reference image to replace a portion of the image area in the first template image to obtain the second template image, or may be to use a specific object contained in the reference image to replace a specific object in the first template image.

[0396] According to an embodiment of the present disclosure, specifically, semantic segmentation can be performed on the first template image to obtain a first object included in the first template image and the position of the first object; and based on the position of the first object, the first object included in the first template image can be replaced with a second object included in the reference image to obtain a second template image.

[0397] According to an embodiment of the present disclosure, correspondingly, the second object contained in the reference image can also be obtained by performing semantic segmentation on the reference image.

[0398] Fig.17A A schematic diagram of a first template image according to an embodiment of the present disclosure is schematically shown.

[0399] Fig. 17B A schematic diagram of a second template image according to an embodiment of the present disclosure is schematically shown.

[0400] like Fig.17A and Fig. 17B As shown, the size of the second object can be adjusted according to the size of the first object by scaling, etc., so that the width and height of the second object are close to the width and height of the first object, and the width and height of the second object are greater than the width and height of the first object. Based on the position of the first object, the second object can be attached to the position of the first object in the first template image by bottom alignment, so that the second object can completely cover the first object, thereby obtaining a second template image.

[0401] According to an embodiment of the present disclosure, the mapping process of pasting the second object onto the first object may cause jagged edges of the second object, so that the second object is partially different from the background. Optionally, after obtaining the second template image, the second template image may be partially redrawn to obtain a third template image, and then the target dynamic image is generated based on the third template image according to the return result of the dynamic image configuration control included in the display interface.

[0402] According to an embodiment of the present disclosure, a third mask image may be generated based on the position of the second object in the second template image.

[0403] According to an embodiment of the present disclosure, optionally, when partially redrawing the second template image, the area to be redrawn may be determined first, and the area to be redrawn may be the area where the second object is located in the second template image. The area to be redrawn may be masked to obtain a third mask image.

[0404] According to an embodiment of the present disclosure, based on the position of the second object in the second template image, the second template image can be binarized to obtain an initial binarized image; the binarized image can be dilated and eroded to obtain a target binarized image; and the target binarized image can be superimposed on the second template image to obtain a third mask image.

[0405] According to an embodiment of the present disclosure, during the binarization process, the pixels in the area where the second object is located in the second template image can be assigned a value of 1, and the pixels in other areas of the second template image can be assigned a value of 0 to obtain an initial binarized image.

[0406] Fig.18A A schematic diagram of a target binarized image according to an embodiment of the present disclosure is schematically shown.

[0407] like Fig.18A As shown, after obtaining the initial binarized image, the pixel points belonging to the area where the second object is located can be dilated respectively to expand the area where the second object is located outward, and the pixel points in the expanded area can be assigned a value of 1, and the area where the second object is located after the expansion is eroded to assign a value of 0 to the pixel points in the partial area that is not related to the edge area of ​​the second object to obtain the target binarized image.

[0408] According to an embodiment of the present disclosure, before superimposing the target binary image onto the second template image, the target binary image may be inverted, that is, multiple pixel points associated with multiple pixel points at the edge of the image belonging to the second object are assigned a value of 0.

[0409] Fig.18B A schematic diagram of a third mask image according to an embodiment of the present disclosure is schematically shown.

[0410] like Fig.18BAs shown, the target binary image after the inversion processing can be superimposed on the second template image. After the inversion processing, only the pixel values ​​of the edge area of ​​the second object are 0, and the values ​​of the pixel points in other areas are all 1, that is, the color of the pixel points in the edge area of ​​the second object is represented as black, and the color of the pixel points in other areas is represented as white. When the target binary image after the inversion processing is superimposed on the second template image, the color of the pixel points in other areas obtained after the superposition can be represented as the color of the pixel points of the original second template image, and the color of the pixel points in the edge area of ​​the second object can be represented as the color of the pixel points of the original target binary image, so that the third mask image can be obtained.

[0411] According to an embodiment of the present disclosure, the second template image may be used as a semantic reference, and image generation may be performed based on the third mask image to obtain a third template image.

[0412] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as a partial redrawing as an example, Fig.14 or Fig.16 The fourth display interface shown may also include an interface jump control "partial redraw" for jumping to the sixth display interface. The sixth display interface may be a display interface when a user performs a task of partially redrawing an image. After the user clicks on the interface jump control, the electronic device may send the generated first template image to the control in the sixth display interface so as to redraw the image using the control in the sixth display interface. Optionally, the reconstructed third template image may also be partially redrawn to obtain a new template image, and then the new template image may be used to generate a new target dynamic image, which will not be described in detail here.

[0413] Fig.19 A schematic diagram of a sixth display interface according to an embodiment of the present disclosure is schematically shown.

[0414] like Fig.19 As shown, similar to the fourth display interface, the sixth display interface may also include a task start control, and the task start control may include a "start generating" button for indicating starting a task.

[0415] According to an embodiment of the present disclosure, the sixth display interface may further include a third image output control, the third image output control may include a fourth window, the fourth window is used to display the image before redrawing, that is, the first template image. The electronic device may send the first template image to the third image output control, and the third image output control may display the first template image in the fourth window on the display interface.

[0416] According to an embodiment of the present disclosure, optionally, the fourth window can be configured as a human-computer interaction window, that is, the user can perform a smear operation on the first template image in the fourth window by touch screen control, mouse control, etc. The user's smear operation can be returned to the electronic device as a return result of the third image output control. The electronic device can determine the target redrawing area in response to the smearing operation on the first template image, and partially redraw the target redrawing area to obtain a fourth template image.

[0417] According to an embodiment of the present disclosure, specifically, in response to a smearing operation on a first template image, a fourth mask image can be generated based on the smearing area indicated by the smearing operation, i.e., the target redrawing area; and using the first template image as a semantic reference, image generation is performed based on the fourth mask image to obtain a fourth template image.

[0418] According to an embodiment of the present disclosure, the sixth display interface may further include a fifth window, and the fifth window may be used to display the redrawn image, that is, the fourth template image.

[0419] According to an embodiment of the present disclosure, the sixth display interface may further include a mode selection configuration control. The mode selection configuration control may be a configuration control of a selection item type. The user may determine the redrawing method of the first template image by selecting from a plurality of selection items included in the mode selection configuration control. For example, the mode selection configuration control may include a selection item representing "erase" and a selection item representing "redraw". When the user selects the selection item representing "erase", the electronic device may erase the foreground elements included in the target redrawing area selected by the user through a redrawing operation. When the user selects the selection item representing "redraw", the electronic device may replace the foreground elements included in the target redrawing area selected by the user with the foreground elements set by the user through a redrawing operation. Optionally, the sixth display interface may further include a redrawing description information input control, which may be a configuration control of an input box type, that is, the redrawing description information input control may include an input box. The user may enter a redrawing description text in the input box, for example, "a little bird" may be entered in the input box. The redrawing description text may be returned to the electronic device as a return result of the redrawing description information input control. The electronic device can obtain the redrawing description text based on the return result of the redrawing description information input control, and after generating a fourth mask image based on the smearing area indicated by the smearing operation, use the first template image and the redrawing description text as semantic references to generate an image based on the fourth mask image to obtain a fourth template image.

[0420] According to an embodiment of the present disclosure, the sixth display interface may also include at least one interface jump control, which can be used to jump from the current sixth display interface to other display interfaces, such as the fourth display interface and the fifth display interface mentioned above, so as to complete subsequent operations on the fourth template image in other display interfaces.

[0421] According to an embodiment of the present disclosure, optionally, the at least one interface jump control may include an interface jump control "dynamic image generation" for jumping to a fifth display interface, and the fifth display interface may be a display interface when a user performs a dynamic image generation task. After the user clicks on the interface jump control, the electronic device may send the generated fourth template image to the control in the fifth display interface so as to generate a dynamic image using the control in the fifth display interface. When generating a dynamic image, the electronic device may generate a target dynamic image based on the fourth template image according to the return result of the dynamic image configuration control contained in the display interface, which will not be elaborated here.

[0422] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as image expansion as an example, Fig.14 or Fig.16 The fourth display interface shown may also include an interface jump control "image expansion" for jumping to the seventh display interface. The seventh display interface may be a display interface when the user performs an image expansion task. After the user clicks on the interface jump control, the electronic device may send the generated first template image to the control in the seventh display interface so as to expand the image using the control in the seventh display interface. Optionally, the image expansion operation may also be performed on the reconstructed third template image, fourth template image, etc. to obtain a new template image, and then the new template image is used to generate a new target dynamic image, which will not be described in detail here.

[0423] Fig. 20 A schematic diagram of a seventh display interface according to an embodiment of the present disclosure is schematically shown.

[0424] like Fig. 20 As shown, similar to the fourth display interface, the seventh display interface may also include a task start control, and the task start control may include a "start generating" button for indicating starting a task and a "restore setting" button for indicating initializing a task.

[0425] According to an embodiment of the present disclosure, the seventh display interface may include a plurality of second configuration controls, and the plurality of second configuration controls may include at least a horizontal resolution configuration control and a vertical resolution configuration control. The horizontal resolution configuration control and the vertical resolution configuration control may both be configuration controls of the input box type. The user may input resolution information in the input box of the horizontal resolution configuration control and the input box of the vertical resolution configuration control, respectively, and the resolution information may be provided to the electronic device as the return result of each of the plurality of second configuration controls. The electronic device may perform image expansion on the first template image based on the resolution information to obtain a fifth template image. For example, the user may enter 1920 in the input box of the horizontal resolution configuration control and 1080 in the input box of the vertical resolution configuration control, and then it may be determined that the resolution information expressed by the user is to adjust the resolution of the image to 1920×1080. The electronic device may perform image expansion on the first template image based on the resolution information representing 1920×1080 to obtain a fifth template image with a resolution of 1920×1080. Specifically, based on the resolution information, a sixth mask image related to the first template image can be generated, and then the first template image is used as a semantic reference to generate an image based on the sixth mask image to obtain a third target mask image, and then the third target mask image is spliced ​​with the first template image to obtain a fifth template image.

[0426] According to an embodiment of the present disclosure, optionally, the seventh display interface may further include a sixth window for displaying the image before expansion, i.e., the first template image, and a seventh window for displaying the image after expansion, i.e., the fifth template image. After completing the image expansion of the first template image, the obtained fifth template image may be displayed in the seventh window to show the fifth template image to the user.

[0427] According to an embodiment of the present disclosure, optionally, the multiple second configuration controls may also include an image style configuration control, which may be a configuration control of a selection item type, and the image style configuration control may include multiple selection items representing image styles, such as a selection item representing an image style of "landscape", a selection item representing an image style of "starry sky", a selection item representing an image style of "poster", etc. The user may determine the image style information by selecting the selection items. The image style information may be used as semantic information to guide the image extension process, that is, in the image extension process, after determining the sixth mask image related to the first template image, the first template image and the image style information may be used as a semantic reference, and image generation may be performed based on the sixth mask image to obtain a third target mask image, and then the third target mask image may be spliced ​​with the first template image to obtain a fifth template image.

[0428] According to an embodiment of the present disclosure, the seventh display interface may also include at least one interface jump control, which can be used to jump from the current seventh display interface to other display interfaces, such as the fourth display interface, the fifth display interface or the sixth display interface mentioned above, so as to complete subsequent operations on the fifth template image in other display interfaces.

[0429] According to an embodiment of the present disclosure, optionally, the at least one interface jump control may include an interface jump control "dynamic image generation" for jumping to the fifth display interface, and the fifth display interface may be a display interface when performing a dynamic image generation task. After the user clicks the interface jump control, the electronic device may send the generated fifth template image to the control in the fifth display interface, so as to generate a dynamic image using the control in the fifth display interface. When generating a dynamic image, the electronic device may generate a target dynamic image based on the fifth template image according to the return result of the dynamic image configuration control contained in the display interface, which is not described in detail here. Optionally, the at least one interface jump control may include an interface jump control "local redraw" for jumping to the sixth display interface, and the sixth display interface may be a display interface when performing a local redraw task. After the user clicks the interface jump control, the electronic device may send the generated fifth template image to the control in the sixth display interface, so as to use the control in the sixth display interface to perform a local redraw of the template image, which is not described in detail here.

[0430] According to an embodiment of the present disclosure, by utilizing the above method for generating a target dynamic image based on a template image, dynamic images of various styles can be generated by fine-tuning the template image, which can effectively reduce the generation cost of dynamic images and improve the flexibility of dynamic image generation.

[0431] Fig.21 The block diagram of a dynamic image generating device according to an embodiment of the present disclosure is schematically shown.

[0432] like Fig.21 As shown, the dynamic image generating device 2100 may include a third acquisition module 2110 , a third image generating module 2120 and a dynamic image generating module 2130 .

[0433] The third acquisition module 2110 is configured to obtain the prompt word text based on the return result of the prompt word input control included in the display interface in response to the second task start control being triggered.

[0434] The third image generation module 2120 is used to generate an image based on the initial template image by taking the prompt word text as a semantic reference to obtain a first template image.

[0435] The dynamic image generation module 2130 is used to generate a target dynamic image based on the first template image according to the return result of the dynamic image configuration control included in the display interface.

[0436] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a first dynamic image generation submodule, a second dynamic image generation submodule, a third dynamic image generation submodule and a fourth dynamic image generation submodule.

[0437] The first dynamic image generation submodule is used to perform depth prediction on the first template image to obtain a depth feature map of the first template image.

[0438] The second dynamic image generation submodule is used to split the first template image into multiple first connected domain images based on the depth feature map.

[0439] The third dynamic image generation submodule is used to generate multiple dynamic image frames based on the return result of the dynamic image configuration control and the multiple first connected domain images.

[0440] The fourth dynamic image generation submodule is used to combine multiple dynamic image frames to obtain a target dynamic image.

[0441] According to an embodiment of the present disclosure, the third dynamic image generation submodule includes a first dynamic image generation unit and a second dynamic image generation unit.

[0442] The first dynamic image generation unit is used to determine the respective processing strategies of the plurality of first connected domain images related to each dynamic image frame according to the return result of the dynamic image configuration control.

[0443] The second dynamic image generation unit is used to process the multiple first connected domain images respectively using respective processing strategies of the multiple first connected domain images related to each dynamic image frame to obtain each dynamic image frame, so as to generate multiple dynamic image frames.

[0444] According to an embodiment of the present disclosure, the second dynamic image generation unit includes a first dynamic image generation sub-unit, a second dynamic image generation sub-unit, and a third dynamic image generation sub-unit.

[0445] The first dynamic image generation subunit is used to translate, fill and / or rotate each first connected domain image based on the processing strategy of each first connected domain image related to each dynamic image frame to obtain a fifth mask image corresponding to each first connected domain image.

[0446] The second dynamic image generation subunit is used to use each first connected domain image as a semantic reference, perform image generation based on a fifth mask image corresponding to each first connected domain image, and obtain multiple second connected domain images related to each dynamic image frame.

[0447] The third dynamic image generation subunit is used to splice a plurality of second connected domain images related to each dynamic image frame to obtain each dynamic image frame.

[0448] According to an embodiment of the present disclosure, the dynamic image generating device 2000 may further include a fourth acquiring module and a fourth image generating module.

[0449] The fourth acquisition module is used to obtain a reference image based on the return result of the second image input control included in the display interface.

[0450] The fourth image generating module is used to generate a second template image based on the first template image and the reference image.

[0451] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a fourth dynamic image generation submodule.

[0452] The fourth dynamic image generation submodule is used to generate a target dynamic image based on the second template image according to the return result of the dynamic image configuration control included in the display interface.

[0453] According to an embodiment of the present disclosure, the fourth image generating module includes a tenth image generating submodule and an eleventh image generating submodule.

[0454] The tenth image generation submodule is used to perform semantic segmentation on the first template image to obtain the first object included in the first template image and the position of the first object.

[0455] The eleventh image generation submodule is used to replace the first object included in the first template image with the second object included in the reference image based on the position of the first object, so as to obtain the second template image.

[0456] According to an embodiment of the present disclosure, the dynamic image generating device 2100 may further include a first redrawing module.

[0457] The first redrawing module is used to partially redraw the second template image to obtain a third template image.

[0458] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a fifth dynamic image generation submodule.

[0459] The fifth dynamic image generation submodule is used to generate a target dynamic image based on the third template image according to the return result of the dynamic image configuration control included in the display interface.

[0460] According to an embodiment of the present disclosure, the first redrawing module includes a first redrawing submodule and a second redrawing submodule.

[0461] The first redrawing submodule is used to generate a third mask image based on the position of the second object in the second template image.

[0462] The second redrawing submodule is used to use the second template image as a semantic reference and generate an image based on the third mask image to obtain a third template image.

[0463] According to an embodiment of the present disclosure, the first redrawing submodule includes a first redrawing unit, a second redrawing unit and a third redrawing unit.

[0464] The first redrawing unit is used to perform binarization processing on the second template image based on the position of the second object in the second template image to obtain an initial binarized image.

[0465] The second redrawing unit is used to perform expansion and corrosion processing on the binary image to obtain a target binary image.

[0466] The third redrawing unit is used to superimpose the target binary image on the second template image to obtain a third mask image.

[0467] According to an embodiment of the present disclosure, the dynamic image generating device 2100 may further include a third sending module and a second redrawing module.

[0468] The third sending module is used to send the first template image to the third image output control, wherein the third image output control is configured to display the first template image on the display interface.

[0469] The second redrawing module is used to partially redraw the first template image in response to the smearing operation on the first template image to obtain a fourth template image.

[0470] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a sixth dynamic image generation submodule.

[0471] The sixth dynamic image generation submodule is used to generate a target dynamic image based on the fourth template image according to the return result of the dynamic image configuration control included in the display interface.

[0472] According to an embodiment of the present disclosure, the second redrawing module includes a third redrawing submodule and a fourth redrawing submodule.

[0473] The third redrawing submodule is used to generate a fourth mask image in response to a smearing operation on the first template image and based on a smearing area indicated by the smearing operation.

[0474] The fourth redrawing submodule is used to use the first template image as a semantic reference and generate an image based on the fourth mask image to obtain a fourth template image.

[0475] According to an embodiment of the present disclosure, the display interface also includes a redraw description information input control.

[0476] According to an embodiment of the present disclosure, the fourth redrawing submodule includes a fourth redrawing unit and a fifth redrawing unit.

[0477] The fourth redrawing unit is used to obtain the redrawing description text based on the return result of the redrawing description information input control.

[0478] The fifth redrawing unit is used to generate an image based on the fourth mask image by taking the first template image and the redrawing description text as semantic references to obtain a fourth template image.

[0479] According to an embodiment of the present disclosure, the dynamic image generating device 2100 may further include a fifth image generating module.

[0480] The fifth image generation module is used to perform image expansion on the first template image based on the return results of each of the multiple second configuration controls included in the display interface to obtain a fifth template image.

[0481] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a seventh dynamic image generation submodule.

[0482] The seventh dynamic image generation submodule is used to generate a target dynamic image based on the fifth template image according to the return result of the dynamic image configuration control included in the display interface.

[0483] According to an embodiment of the present disclosure, the third acquisition module 2110 includes a third acquisition submodule and a fourth acquisition submodule.

[0484] The third acquisition submodule is used to obtain multiple prompt words based on the return result of the prompt word input control.

[0485] The fourth acquisition submodule is used to write multiple prompt words into the template text respectively to obtain the prompt word text.

[0486] It should be noted that the dynamic image generating device part in the embodiment of the present disclosure corresponds to the dynamic image generating method part in the embodiment of the present disclosure. The description of the dynamic image generating device part specifically refers to the dynamic image generating method part, which will not be repeated here.

[0487] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0488] For example, any multiple of the third acquisition module 2110, the third image generation module 2120, and the dynamic image generation module 2130 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more modules / units / subunits in these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the third acquisition module 2110, the third image generation module 2120, and the dynamic image generation module 2130 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the third acquisition module 2110, the third image generation module 2120 and the dynamic image generation module 2130 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed. Fig. 22 A block diagram of an electronic device suitable for implementing an image processing method or a dynamic image generating method according to an embodiment of the present disclosure is schematically shown. Fig. 22 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0489] like Fig. 22 As shown, the computer electronic device 2200 according to the embodiment of the present disclosure includes a processor 2201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2202 or the program loaded from the storage part 2208 to the random access memory (RAM) 2203. The processor 2201 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 2201 may also include an onboard memory for caching purposes. The processor 2201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0490] In RAM 2203, various programs and data required for the operation of electronic device 2200 are stored. Processor 2201, ROM 2202 and RAM 2203 are connected to each other through bus 2204. Processor 2201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 2202 and / or RAM 2203. It should be noted that the program can also be stored in one or more memories other than ROM 2202 and RAM 2203. Processor 2201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0491] According to an embodiment of the present disclosure, the electronic device 2200 may further include an input / output (I / O) interface 2205, which is also connected to the bus 2204. The electronic device 2200 may further include one or more of the following components connected to the input / output (I / O) interface 2205: an input portion 2206 including a keyboard, a mouse, etc.; an output portion 2207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 2208 including a hard disk, etc.; and a communication portion 2209 including a network interface card such as a LAN card, a modem, etc. The communication portion 2209 performs communication processing via a network such as the Internet. A drive 22222210 is also connected to the input / output (I / O) interface 2205 as needed. A removable medium 2211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 22222210 as needed, so that a computer program read therefrom is installed into the storage portion 2208 as needed.

[0492] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2209, and / or installed from the removable medium 2211. When the computer program is executed by the processor 2201, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0493] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0494] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0495] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 2202 and / or the RAM 2203 described above and / or one or more memories other than the ROM 2202 and the RAM 2203 .

[0496] The embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiments of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiments of the present disclosure.

[0497] When the computer program is executed by the processor 2201, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0498] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 2209, and / or installed from the removable medium 2211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0499] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0500] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0501] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method, comprising: In response to the image expansion request, performing visual language recognition on the first original image to obtain image description text; Generate a second original image and a first mask image based on the first original image using the target aspect ratio and the first target resolution included in the image expansion request; Using the image description text and the second original image as semantic references, and performing image generation based on the first mask image to obtain a first target mask image; as well as The first target mask image and the second original image are stitched together to obtain a target extended image.

2. The method according to claim 1, wherein The step of generating a second original image and a first mask image based on the first original image by utilizing the target aspect ratio and the first target resolution included in the image extension request includes: Based on the target aspect ratio and the first target resolution, performing image expansion with at least one side of the first original image as a starting point to obtain a first expanded image; and The second original image and the first mask image are determined from the first extended image, wherein the second original image is the first original image.

3. The method according to claim 1, wherein The step of generating a second original image and a first mask image based on the first original image by utilizing the target aspect ratio and the first target resolution included in the image extension request includes: performing scaling processing on the first original image based on an original resolution and a preset resolution of the first original image to obtain a third original image; Based on the target aspect ratio, performing image expansion with at least one side of the third original image as a starting point to obtain a second expanded image; and When the first target resolution is equal to the preset resolution, the second original image and the first mask image are determined from the second extended image.

4. The method according to claim 3, further comprising: When the first target resolution is not equal to the preset resolution, scaling the first extended image based on the first target resolution and the preset resolution to obtain a third extended image; as well as The second original image and the first mask image are determined from the third extended image.

5. The method according to any one of claims 1 to 4, further comprising: When it is determined based on the first target resolution and the original resolution of the first original image that a preset condition is satisfied, performing N image expansions using the first original image and the image description text as semantic references to obtain the target expanded image, where N is a positive integer; The preset conditions include any one of the following: A ratio between the first target resolution and the original resolution of the first original image is less than a first threshold; a ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold; and The image extension request includes an image extension count greater than 1.

6. The method according to claim 5, wherein: The step of performing N image expansions using the first original image and the image description text as semantic references to obtain the target expanded image includes: determining an expansion ratio based on the first target resolution, the original resolution of the first original image, and N; In a case where the first target resolution and the image resolution of the extended image after the (i-1)th image expansion are not equal, determining a second target resolution based on the image resolution of the extended image after the (i-1)th image expansion and the expansion magnification, wherein i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution; generating a fourth original image and a second mask image based on the extended image after the (i-1)th image expansion by using the target aspect ratio and the second target resolution; Using the image description text and the fourth original image as semantic references, performing image generation based on the second mask image to obtain a second target mask image; and The fourth original image and the second target mask image are stitched together to obtain an expanded image after the i-th image expansion.

7. The method according to claim 6, further comprising: In a case where the first target resolution is equal to the image resolution of the extended image after the i-th image expansion, the extended image after the i-th image expansion is determined as the target extended image.

8. The method according to claim 1, wherein The step of generating an image based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image includes: Using a multimodal encoder to encode the second original image and the image description text respectively to obtain image encoding features and text encoding features; Initializing the first mask image using Gaussian noise to obtain a noise image; Using the image coding features and text coding features as semantic benchmarks, processing the noisy image using the image information creator to obtain target output features; and The target output features are decoded using a model decoder to obtain the first target mask image.

9. The method according to claim 8, wherein The image information creator includes an encoding module, a decoding module and an intermediate module; The step of using the image coding features and the text coding features as semantic benchmarks and processing the noise image using the image information creator to obtain target output features includes: Processing the image coding feature, the text coding feature, and the noise image using the coding module to obtain first noise image features output by each of M coding function blocks included in the coding module, where M is a positive integer; Processing the first noise image feature output by the Mth encoding function block using the intermediate module to obtain a second noise image feature; and The decoding module is used to process the second noise image feature, the first noise image feature output by each of the M encoding function blocks, the image coding feature and the text coding feature to obtain the target output feature.

10. The method according to claim 9, wherein: The M encoding function blocks are connected by cross attention blocks; The step of processing the image coding feature, the text coding feature, and the noise image using the coding module to obtain first noise image features output by respective M coding function blocks included in the coding module includes: Processing the input feature of the j-th encoding function block using the j-th encoding function block to obtain a first noise image feature output by the j-th encoding function block, wherein j is a positive integer less than or equal to M, and the input feature of the first encoding function block is represented as the noise image; and When j is less than M, the cross-attention block between the j-th encoding functional block and the j+1-th encoding functional block is used to process the first noise image feature, the image encoding feature, and the text encoding feature output by the j-th encoding functional block to obtain the input feature of the j+1-th encoding functional block.

11. The method according to claim 9, wherein The decoding module includes M decoding function blocks, and the M decoding function blocks are connected by cross attention blocks; The step of using the decoding module to process the second noise image feature, the first noise image feature outputted by each of the M encoding function blocks, the image encoding feature, and the text encoding feature to obtain the target output feature includes: Processing the input feature of the kth decoding function block and the first noise image feature output by the kth encoding function block using the kth decoding function block to obtain a decoding feature output by the kth decoding function block, wherein k is a positive integer less than or equal to M, and the input feature of the first decoding function block is represented as the second noise image feature; When k is less than M, using a cross attention block between the kth decoding functional block and the k+1th decoding functional block to process the decoding features output by the kth decoding functional block, the image coding features, and the text coding features, to obtain input features of the k+1th decoding functional block; and When k is equal to M, the target output feature is determined to be the decoding feature output by the kth decoding function block.

12. The method according to any one of claims 9 to 11, wherein The intermediate module, the encoding function block and the decoding function block are gated convolution blocks; The step of processing the first noise image feature output by the Mth encoding function block using the intermediate module to obtain the second noise image feature includes: Performing layer normalization processing on the first noise image feature output by the M-th encoding function block to obtain a first sub-feature; Processing the first sub-feature using a gated convolutional layer to obtain a second sub-feature; Performing layer normalization on the second sub-feature to obtain a third sub-feature; Processing the third sub-feature using a perceptron to obtain a fourth sub-feature; and The second noise image feature is obtained based on the first noise image feature output by the Mth encoding function block and the fourth sub-feature.

13. The method according to claim 12, wherein: The gated convolution layer includes P gated processing units, where P is a positive integer; The step of processing the first sub-feature using a gated convolutional layer to obtain the second sub-feature includes: Performing convolution processing on the first sub-feature to obtain a fifth sub-feature; Performing depth convolution processing on the fifth sub-feature to obtain a depth convolution feature; For the p-th gated processing unit, performing a dot product process on the depth convolution feature and the gated output feature output by the p-1-th gated processing unit to obtain a sixth sub-feature of the p-th gated processing unit, where p is a positive integer less than or equal to P; and Convolution processing is performed on the sixth sub-feature of the p-th gated processing unit to obtain a gated output feature output by the p-th gated processing unit, wherein the gated output feature of the 0th gated processing unit is represented as the fifth sub-feature, and the gated output feature of the P-th gated processing unit is represented as the second sub-feature.

14. The method according to claim 1, wherein The performing visual language recognition on the first original image to obtain image description text includes: Processing the first original image using a visual language pre-trained model to obtain at least one object text; and The image description text is obtained based on the at least one object text.

15. The method according to claim 14, wherein The image extension request further includes an image extension keyword; The method further comprises: The image description text is obtained based on the at least one object text and the image extension keyword.

16. An image processing method, comprising: In response to the first task start control being triggered, obtaining a first original image from a return result of a first image input control included in the display interface; Performing visual language recognition on the first original image to obtain image description text; Obtaining a target aspect ratio and a first target resolution based on return results of each of the plurality of first configuration controls included in the display interface; generating a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution; Using the image description text and the second original image as semantic references, and performing image generation based on the first mask image to obtain a first target mask image; as well as The first target mask image and the second original image are stitched together to obtain a target extended image.

17. The method according to claim 16, wherein The plurality of first configuration controls include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control; The obtaining of the target aspect ratio and the first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface includes: Determining the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control; and The target aspect ratio is determined based on the first target resolution.

18. The method according to claim 16, wherein The plurality of first configuration controls further include an expansion direction configuration control; The step of generating a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution includes: determining at least one target edge from the four edges of the first original image based on the expansion direction information returned by the expansion direction configuration control; Based on the target aspect ratio and the first target resolution, performing image expansion with the at least one target edge of the first original image as a starting point to obtain a first expanded image; and The second original image and the first mask image are determined from the first extended image, wherein the second original image is the first original image.

19. The method according to claim 18, wherein The plurality of first configuration controls further includes a candidate image quantity control; The step of performing image expansion based on the target aspect ratio and the first target resolution and taking the at least one target edge of the first original image as a starting point to obtain a first expanded image includes: Determining the number of candidate images based on the candidate image number information returned by the candidate image number control; Based on the target aspect ratio and the first target resolution, performing image expansion with the at least one target edge of the first original image as a starting point to obtain fifth expanded images equal to the number of candidate images; Sending a fifth expanded image of the number of candidate images to a first image output control, wherein the first image output control is configured to display the fifth expanded image of the number of candidate images on the display interface; and In response to a selection request for a fifth extended image of the candidate images, the first extended image is determined from the fifth extended images of the candidate images.

20. The method according to claim 16, wherein The plurality of first configuration controls further include a generation speed configuration control and a single maximum expansion pixel control; The method further comprises: Determining the number of image expansion times based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image, and the first target resolution; and When the number of image expansion times is greater than 1, multiple image expansions are performed based on the number of image expansion times with the first original image and the image description text as semantic references to obtain the target expanded image.

21. The method according to claim 16, wherein The plurality of first configuration controls further include an image style configuration control; The performing visual language recognition on the first original image to obtain image description text includes: Obtaining image extension keywords based on the image style information returned by the image style configuration control; Processing the first original image using a visual language pre-trained model to obtain at least one object text; and The image description text is obtained based on the at least one object text and the image extension keyword.

22. The method according to claim 16, wherein The plurality of first configuration controls further includes a creativity configuration control; The step of using the image description text and the second original image as semantic references and performing image generation based on the first mask image to obtain a first target mask image includes: Using a multimodal encoder to encode the second original image and the image description text respectively to obtain initial image encoding features and initial text encoding features; Based on the creativity information returned by the creativity configuration control, the initial image coding feature and the initial text coding feature are modified to obtain an image coding feature and a text coding feature; Initializing the first mask image using Gaussian noise to obtain a noise image; Using the image coding features and text coding features as semantic benchmarks, processing the noisy image using the image information creator to obtain target output features; and The target output features are decoded using a model decoder to obtain the first target mask image.

23. The method of claim 16, further comprising: determining resolution information of the first original image in response to an image input request associated with the first original image; as well as The resolution information is sent to a resolution display control, wherein the resolution display control is configured to display the original resolution of the first original image on the display interface based on the resolution information.

24. The method of claim 16, further comprising: The target expanded image is sent to a second image output control, wherein the second image output control is configured to display the target expanded image on the display interface.

25. An image processing device comprising: A first recognition module is configured to perform visual language recognition on the first original image in response to the image expansion request to obtain image description text; a first generating module, configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution included in the image extension request; a first image generation module, configured to generate an image based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image; as well as The first stitching module is configured to stitch the first target mask image and the second original image to obtain a target extended image.

26. An image processing apparatus, comprising: A first acquisition module is configured to acquire a first original image from a return result of a first image input control included in the display interface in response to the first task start control being triggered; A second recognition module is used to perform visual language recognition on the first original image to obtain image description text; A second acquisition module, configured to obtain a target aspect ratio and a first target resolution based on return results of each of the plurality of first configuration controls included in the display interface; a second generating module, configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution; a second image generation module, configured to generate an image based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image; as well as The second stitching module is configured to stitch the first target mask image and the second original image to obtain a target extended image.

27. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 24.

28. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 24.

29. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 24 when executed.

Images