Image processing method and apparatus, and electronic device and storage medium

Abstract

The present disclosure relates to the technical field of artificial intelligence, and particularly relates to the technical fields of computer vision, image processing and machine learning. Provided are an image processing method and apparatus, and an electronic device and a storage medium. The image processing method comprises: in response to an image extension request, performing visual language recognition on a first original image to obtain image description text; using a target aspect ratio and a first target resolution, which are included in the image extension request, to generate a second original image and a first mask image on the basis of the first original image; by taking the image description text and the second original image as a semantic reference, performing image generation on the basis of the first mask image to obtain a first target mask image; and performing image stitching on the first target mask image and the second original image to obtain a target extended image.

Description

Image processing method, device, electronic device and storage medium

[0001] This disclosure requests the priority of invention patent application with application number 202310899530.3, application date July 20, 2023, and invention title "Image processing method, device, electronic device and storage medium"; and requests the priority of invention patent application with application number 202410178176.X, application date February 8, 2024, and invention title "Image processing method, device, electronic device and storage medium"; the contents of the priority text have been fully cited in this disclosure. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision, image processing, and machine learning technology, and more specifically to an image processing method, device, electronic device, and storage medium. Background Art

[0003] Image extension technology is an image generation technology that aims to understand the semantic content and structure of an image and generate an extended part that is consistent with the style and content of the original image, so as to expand the boundaries or canvas of the image and generate a larger or more complete image. It has wide applications in image editing, virtual reality and other fields.

[0004] Summary of the Invention

[0005] In view of this, the present disclosure provides an image processing method, apparatus, electronic device, readable storage medium, and computer program product.

[0006] One aspect of the present disclosure provides an image processing method, comprising: in response to an image expansion request, performing visual language recognition on a first original image to obtain image description text; using the target aspect ratio and first target resolution included in the above-mentioned image expansion request, based on the above-mentioned first original image, generating a second original image and a first mask image; using the above-mentioned image description text and the above-mentioned second original image as semantic references, performing image generation based on the above-mentioned first mask image to obtain a first target mask image; and performing image splicing on the above-mentioned first target mask image and the above-mentioned second original image to obtain a target expanded image.

[0007] Another aspect of the present disclosure provides an image processing method, including: in response to a first task start control being triggered, obtaining a first original image from the return result of a first image input control included in a display interface; performing visual language recognition on the above-mentioned first original image to obtain image description text; based on the return results of each of multiple first configuration controls included in the above-mentioned display interface, obtaining a target aspect ratio and a first target resolution; using the above-mentioned target aspect ratio and the above-mentioned first target resolution, based on the above-mentioned first original image, generating a second original image and a first mask image; using the above-mentioned image description text and the above-mentioned second original image as semantic references, performing image generation based on the above-mentioned first mask image to obtain a first target mask image; and performing image splicing on the above-mentioned first target mask image and the above-mentioned second original image to obtain a target extended image.

[0008] Another aspect of the present disclosure provides an image processing device, including: a first recognition module, used to perform visual language recognition on a first original image in response to an image expansion request to obtain image description text; a first generation module, used to generate a second original image and a first mask image based on the first original image using the target aspect ratio and first target resolution included in the above-mentioned image expansion request; a first image generation module, used to use the above-mentioned image description text and the above-mentioned second original image as semantic references, perform image generation based on the above-mentioned first mask image to obtain a first target mask image; and a first stitching module, used to stitch the above-mentioned first target mask image with the above-mentioned second original image to obtain a target expanded image.

[0009] Another aspect of the present disclosure provides an image processing device, including: a first acquisition module, for acquiring a first original image from the return result of a first image input control included in a display interface in response to a first task start control being triggered; a second recognition module, for performing visual language recognition on the above-mentioned first original image to obtain image description text; a second acquisition module, for obtaining a target aspect ratio and a first target resolution based on the return results of each of a plurality of first configuration controls included in the above-mentioned display interface; a second generation module, for generating a second original image and a first mask image based on the above-mentioned first original image using the above-mentioned target aspect ratio and the above-mentioned first target resolution; a second image generation module, for performing image generation based on the above-mentioned first mask image using the above-mentioned image description text and the above-mentioned second original image as semantic references to obtain a first target mask image; and a second stitching module, for stitching the above-mentioned first target mask image with the above-mentioned second original image to obtain a target extended image.

[0010] Another aspect of the present disclosure provides a method for generating a dynamic image, comprising: in response to a second task start control being triggered, obtaining a prompt word text based on the return result of a prompt word input control included in a display interface; using the above prompt word text as a semantic reference, performing image generation based on an initial template image to obtain a first template image; and generating a target dynamic image based on the above first template image according to the return result of a dynamic image configuration control included in the above display interface.

[0011] Another aspect of the present disclosure provides a dynamic image generation device, including: a third acquisition module, used to obtain a prompt word text based on the return result of the prompt word input control included in the display interface in response to the second task start control being triggered; a third image generation module, used to use the above-mentioned prompt word text as a semantic reference, generate an image based on the initial template image, and obtain a first template image; and a dynamic image generation module, used to generate a target dynamic image based on the above-mentioned first template image according to the return result of the dynamic image configuration control included in the above-mentioned display interface.

[0012] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0013] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0014] Another aspect of the present disclosure provides a computer program product, which includes computer-executable instructions. When the instructions are executed, they are used to implement the method described above.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] FIG1 schematically shows an exemplary system architecture to which the image processing method and apparatus according to an embodiment of the present disclosure can be applied.

[0018] FIG2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.

[0019] FIG3A schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to an embodiment of the present disclosure.

[0020] FIG3B schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to another embodiment of the present disclosure.

[0021] FIG3C schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to yet another embodiment of the present disclosure.

[0022] FIG4 schematically shows a flow chart of generating a target extended image.

[0023] FIG5 schematically shows a structural diagram of an image information creator according to an embodiment of the present disclosure.

[0024] FIG6A schematically shows a structural diagram of an intermediate module according to an embodiment of the present disclosure.

[0025] FIG6B schematically shows a structural diagram of an intermediate module according to another embodiment of the present disclosure.

[0026] FIG6C schematically shows a structural diagram of a gated convolutional layer according to another embodiment of the present disclosure.

[0027] FIG7 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.

[0028] FIG8A schematically shows a schematic diagram of a first original image according to another embodiment of the present disclosure.

[0029] FIG8B schematically shows a schematic diagram of an image after one expansion according to another embodiment of the present disclosure.

[0030] FIG8C schematically shows a schematic diagram of a target extended image according to another embodiment of the present disclosure.

[0031] FIG9 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.

[0032] FIG10A schematically shows a schematic diagram of a first display interface according to an embodiment of the present disclosure.

[0033] FIG10B schematically shows a schematic diagram of a second display interface according to an embodiment of the present disclosure.

[0034] FIG10C schematically shows a schematic diagram of a third display interface according to an embodiment of the present disclosure.

[0035] FIG11 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0036] FIG12 schematically shows a block diagram of an image processing apparatus according to another embodiment of the present disclosure.

[0037] FIG13 schematically shows a flow chart of a method for generating a dynamic image according to an embodiment of the present disclosure.

[0038] FIG14 schematically shows a schematic diagram of a fourth display interface according to an embodiment of the present disclosure.

[0039] FIG15 schematically shows a schematic diagram of a fifth display interface according to an embodiment of the present disclosure.

[0040] FIG16 schematically shows a schematic diagram of a fourth display interface according to another embodiment of the present disclosure.

[0041] FIG17A schematically shows a schematic diagram of a first template image according to an embodiment of the present disclosure.

[0042] FIG17B schematically shows a schematic diagram of a second template image according to an embodiment of the present disclosure.

[0043] FIG18A schematically shows a schematic diagram of a target binarized image according to an embodiment of the present disclosure.

[0044] FIG18B schematically shows a schematic diagram of a third mask image according to an embodiment of the present disclosure.

[0045] FIG19 schematically shows a schematic diagram of a sixth display interface according to an embodiment of the present disclosure.

[0046] FIG20 schematically shows a schematic diagram of a seventh display interface according to an embodiment of the present disclosure.

[0047] FIG21 schematically shows a block diagram of a dynamic image generating apparatus according to an embodiment of the present disclosure.

[0048] FIG22 schematically shows a block diagram of an electronic device suitable for implementing an image processing method or a dynamic image generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0050] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0051] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0052] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0053] Embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium. The method includes: in response to an image expansion request, performing visual language recognition on a first original image to obtain image description text; generating a second original image and a first mask image based on the first original image using a target aspect ratio and a first target resolution included in the image expansion request; performing image generation based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image; and performing image splicing on the first target mask image and the second original image to obtain a target expanded image.

[0054] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of the data involved (for example, including but not limited to user personal information) all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0055] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0056] Figure 1 schematically illustrates an exemplary system architecture to which the image processing method and apparatus according to an embodiment of the present disclosure may be applied. It should be noted that Figure 1 merely illustrates an example of a system architecture to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, and does not imply that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios.

[0057] As shown in FIG. 1 , the system architecture 100 according to this embodiment may include terminal devices 101 and 102 , a network 103 and a server 104 .

[0058] The terminal devices 101 and 102 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. Optionally, the resolutions of the display screens of the terminal devices 101 and 102 may be different.

[0059] The network 103 is used to provide a medium for communication links between the terminal devices 101 and 102 and the server 104. The network 103 may include various connection types, such as wired and / or wireless communication links, etc.

[0060] The server 104 can be a local server that provides various services, or it can be various cloud servers, which is not limited here.

[0061] It should be noted that the image processing method provided in the embodiment of the present disclosure can generally be executed by the server 104. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can generally be set in the server 104. The image processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 104 and can communicate with the terminal devices 101, 102 and / or the server 104. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 104 and can communicate with the terminal devices 101, 102 and / or the server 104. Alternatively, the image processing method provided in the embodiment of the present disclosure can also be executed by the terminal device 101 or 102, or can also be executed by other terminal devices different from the terminal device 101 or 102. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101 or 102, or can be set in other terminal devices different from the terminal device 101 or 102.

[0062] For example, a user may generate an image expansion request by operating terminal device 101. This image expansion request instructs that a first original image originally adapted for the display screen of terminal device 101 be expanded into a target expanded image adapted for the display screen of terminal device 102. Terminal device 101 may send the first original image and the image expansion request to server 104 via network 103. Server 104 may respond to the image expansion request, execute the image processing method according to an embodiment of the present disclosure to process the first original image, obtain the target expanded image, and send the target expanded image to terminal device 102 via network 103. After receiving the target expanded image, terminal device 102 may display the target expanded image on its display screen.

[0063] It should be understood that the number of terminal devices, networks, and servers in Figure 1 is merely illustrative and any number of terminal devices, networks, and servers may be provided as required.

[0064] FIG2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.

[0065] As shown in FIG. 2 , the method 200 includes operations S210 to S240 .

[0066] In operation S210 , in response to an image expansion request, visual language recognition is performed on the first original image to obtain image description text.

[0067] In operation S220, a second original image and a first mask image are generated based on the first original image using the target aspect ratio and the first target resolution included in the image expansion request.

[0068] In operation S230 , an image is generated based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image.

[0069] In operation S240 , the first target mask image and the second original image are spliced ​​to obtain a target extended image.

[0070] According to an embodiment of the present disclosure, the first original image may be the image on which the image is expanded, that is, the image style and content on which the image is expanded may be the image style and content of the first original image.

[0071] According to an embodiment of the present disclosure, performing visual language recognition on the first original image may include performing semantic analysis on the first original image, and the resulting image description text may include semantic information of the first original image. Alternatively, performing visual language recognition on the first original image may include identifying various objects included in the first original image, and the resulting image description text may include attribute information of the various objects.

[0072] According to an embodiment of the present disclosure, any vision-language pre-trained (VLP) model can be used to perform vision-language recognition on the first original image. The vision-language pre-training model may include a multimodal learning model based on images and text, a multi-label classification model based on images and text, etc., which is not limited here.

[0073] According to an embodiment of the present disclosure, the target aspect ratio and the first target resolution can be respectively expressed as the aspect ratio and resolution of the target extended image to be generated. The target aspect ratio and the first target resolution can be directly input into the terminal device by the user, or the user can also input relevant image extension information. The target aspect ratio and the first target resolution can be calculated based on the relevant image extension information input into the terminal device by the user. For example, in a multi-screen extension scenario, the relevant image extension information input into the terminal device by the user may include the number of horizontal screens, the number of vertical screens, and the resolution of a single screen. Based on the number of horizontal screens, the number of vertical screens, and the resolution of a single screen, the target aspect ratio and the first target resolution can be calculated. Taking the example of 2 horizontal screens, 3 vertical screens, and a single screen resolution of 1920×1080, the first target resolution that can be calculated is (1920×2)×(1080×3)=3840×3840, and then based on the first target resolution, the aspect ratio can be calculated to be 3840:3840=1:1.

[0074] According to an embodiment of the present disclosure, the second original image may be the first original image after resolution adjustment based on the first target resolution. That is, the long side resolution of the second original image may be equal to the long side resolution of the target expanded image, or the short side resolution of the second original image may be equal to the short side resolution of the target expanded image. The resolution scaling operation on the first original image may include downsampling when reducing the resolution, interpolation when increasing the resolution, and the like.

[0075] According to an embodiment of the present disclosure, the first mask image may be an image of an extended canvas portion. The canvas of the second original image is spliced ​​with the canvas of the first mask image to obtain a canvas of the target extended image to be generated.

[0076] According to an embodiment of the present disclosure, the data of each pixel in the image data of the generated first mask image may be any value.

[0077] According to an embodiment of the present disclosure, the image description text and the semantic information contained in the second original image can be used to guide the first mask image for image generation, so that an image with the same content or style is generated in the first mask image portion, thereby obtaining a first target mask image.

[0078] According to an embodiment of the present disclosure, the image of the expanded canvas portion may be replaced by the first target mask image from the first mask image to achieve splicing of the first target mask image and the second original image.

[0079] According to an embodiment of the present disclosure, when performing image expansion, an image description text can be generated based on the semantic information of the first original image, a second original image can be generated based on the first original image, and a first mask image can be expanded. The image description text and the semantic information contained in the second original image are then used to guide the first mask image for image generation to obtain a first target mask image, so that the generated first target mask image can have the same or similar content and style as the first original image. The first target mask image can be spliced ​​with the second original image to obtain a target extended image, which can effectively improve the quality of the generated target extended image.

[0080] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations, or there is a sequence of execution between different operations in terms of technical implementation, the operations shown in the flowchart of the embodiment of the present disclosure may be executed in any order, and multiple operations may be executed simultaneously. For example, after receiving an image expansion request, operation S210 may be executed first to generate image description text, and then operation S220 may be executed to expand the first original image; alternatively, operation S220 may be executed first to expand the first original image, and then operation S210 may be executed to generate image description text; alternatively, operations S210 and S220 may be executed simultaneously.

[0081] The method shown in FIG. 2 will be further described below in conjunction with specific embodiments with reference to FIG. 3A to FIG. 3C , FIG. 4 , FIG. 5 , FIG. 6A to FIG. 6C , FIG. 7 , and FIG. 8A to FIG. 8C .

[0082] According to an embodiment of the present disclosure, a user may input a target aspect ratio and a first target resolution on a terminal device, and click a button control indicating “start generating” to enable the terminal device to generate an image expansion request.

[0083] According to an embodiment of the present disclosure, in response to an image expansion request, the first original image can be processed using a visual language pre-trained model to obtain at least one object text; and then based on the at least one object text, an image description text is obtained, that is, text used to describe the first original image.

[0084] According to an embodiment of the present disclosure, the visual language pre-training model may be a CLIP (Contrastive Language-Image Pretraining) model. The CLIP model can achieve semantic understanding and alignment between images and texts. The CLIP model can use a contrastive learning method and a large-scale image and text dataset for pre-training. It can learn to bring the embedding vectors of related image and text pairs close to each other in the semantic space, while separating the embedding vectors of unrelated image and text pairs from each other in the semantic space, so that it has cross-modal capabilities and can link images and texts. When the CLIP model is applied, one or more matching texts, i.e., at least one object text, can be determined from a plurality of preset texts by matching the embedding vector of the input image with the respective embedding vectors of a plurality of preset texts. The at least one object text can be spliced ​​to obtain an image description text.

[0085] According to an embodiment of the present disclosure, the visual language pre-training model may also be a multi-label classification model, which may identify objects such as scenes, objects, and people in the first original image to obtain at least one object label. The name of each of the at least one object label may be used as at least one object text, and the at least one object text may be spliced ​​to obtain an image description text. Alternatively, the name of each of the at least one object label may be respectively entered into a text template to obtain at least one object text, and the at least one object text may be spliced ​​to obtain an image description text. Alternatively, the name of each of the at least one object label may be entered into a text template to directly obtain the image description text.

[0086] According to an embodiment of the present disclosure, as an optional implementation, for specific scenarios, such as when the first original image is a poster-like or landscape-like image, or when the desired image expansion content is specific to a particular scene, developers can preset certain keywords. For example, for poster-like images, preset keywords may include "poster," "gradient background," "widescreen," "illustration," and "vector." For landscape-like images, preset keywords may include "realistic," "film texture," "photo-like," and "high quality." Before generating an image expansion request, a user can select an image expansion keyword from a preset list of keywords, and the image expansion request can accordingly include the image expansion keyword. When generating image description text, the image description text can be obtained based on at least one object text and the image expansion keyword. Specifically, the at least one object text and the image expansion keyword can be concatenated to generate the image description text. Alternatively, the name of each of the at least one object tag and the image expansion keyword can be separately entered into a text template to generate at least one object text, and the at least one object text can be concatenated to generate the image description text. Alternatively, the name of each of the at least one object tag and the image expansion keyword can be entered into a text template to directly generate the image description text.

[0087] According to an embodiment of the present disclosure, while generating image description text, the first original image can be expanded. Image expansion can be performed directly based on the first original image. Specifically, the target aspect ratio and the first target resolution included in the image expansion request are used to generate a second original image and a first mask image based on the first original image. The following operations can be performed:

[0088] Based on the target aspect ratio and the first target resolution, image expansion is performed with at least one side of the first original image as a starting point to obtain a first extended image; and a second original image and a first mask image are determined from the first extended image.

[0089] According to an embodiment of the present disclosure, image expansion is performed with at least one side of the first original image as a starting point, which may refer to image expansion with at least one side of the first original image as a common side. For example, the first original image may be a rectangular image surrounded by side A, side B, side C, and side D. When image expansion is performed with side A as a starting point, side B may be extended from the intersection of side B and side A, and the extended portion may be considered as side B'. Side D may be extended from the intersection of side D and side A, and the extended portion may be considered as side D'. The extended end points of sides B' and D' are connected to obtain a new side, which may be considered as side A'. After image expansion is performed with side A as a starting point, the first extended image obtained may be composed of side A', side B, side B', side C, side D, and side D', wherein side A', side B', side A, and side D' constitute the first mask image.

[0090] According to an embodiment of the present disclosure, the second original image determined from the first extended image is the first original image.

[0091] FIG3A schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to an embodiment of the present disclosure.

[0092] As shown in FIG3A , the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, and the first target resolution may be 2048×2048. Based on the first target resolution and the target aspect ratio, the first original image 301 may be expanded to obtain a first expanded image 302 having a resolution of 2048×2048. A second original image 303 may be determined in the first expanded image 302 to be the first original image 301, and the portion of the first expanded image 302 excluding the first original image 301 may be a first mask image 304.

[0093] According to an embodiment of the present disclosure, an image expansion is performed directly on the basis of the first original image. When the first target resolution is much larger than the original resolution, the second original image obtained after scaling has a larger size. Image expansion based on the second original image will consume more computing resources. Therefore, as an optional implementation method, in order to reduce the size of the image during calculation and improve the calculation efficiency, the first original image can be first reduced in size, and after the image expansion is completed, the expanded image can be enlarged to the required resolution.

[0094] According to an embodiment of the present disclosure, when the first original image is reduced, the reduction process can be performed based on a preset resolution. The preset resolution can be represented as a smaller resolution, and the preset resolution can include only the resolution of the long side or the resolution of the short side. The preset resolution can be set by the developer, or the preset resolution can also be obtained through user input, which is not limited here.

[0095] According to embodiments of the present disclosure, in some embodiments, the preset resolution may be equal to the first target resolution. In this case, after image expansion is completed, the desired image can be obtained. Specifically, using the target aspect ratio and the first target resolution included in the image expansion request, generating the second original image and the first mask image based on the first original image may include the following operations:

[0096] Based on the original resolution and the preset resolution of the first original image, the first original image is scaled to obtain a third original image; based on the target aspect ratio, the image is expanded with at least one side of the third original image as a starting point to obtain a second expanded image; when the first target resolution is equal to the preset resolution, the second original image and the first mask image are determined from the second expanded image.

[0097] According to an embodiment of the present disclosure, when the original resolution is greater than a preset resolution, a downsampling process may be performed on the first original image to reduce the resolution of the first original image from the original resolution to the preset resolution. The downsampling process may include a maximum pooling process, an average pooling process, and the like.

[0098] According to an embodiment of the present disclosure, when the original resolution is less than the preset resolution, the first original image can be interpolated, or the first original image can be processed using a super-resolution algorithm to increase the resolution of the first original image from the original resolution to the preset resolution. The interpolation process may include bilinear interpolation, nearest neighbor interpolation, etc. The super-resolution algorithm may include an ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) method based on GAN (Generative Adversarial Networks), a transformer-based SwinIR (Image Restoration Using Swin Transformer) method, etc.

[0099] According to an embodiment of the present disclosure, when the original resolution is equal to the preset resolution, the second original image may be determined to be the first original image.

[0100] According to an embodiment of the present disclosure, when performing scaling, the long side resolution and the short side resolution of the first original image can be proportionally scaled to the long side resolution or the short side resolution represented by the preset resolution based on the long side or the short side resolution of the first original image, so that the long side resolution of the third original image obtained after scaling is equal to the long side resolution represented by the preset resolution, or the short side resolution of the third original image is equal to the short side resolution represented by the preset resolution. For example, the resolution of the first original image can be 1024×512 and the preset resolution can be 2048×2048. Then, based on the long side of the first original image, the resolution of the first original image can be scaled up by 2 times, and the resolution of the third original image can be 2048×1024.

[0101] FIG3B schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to another embodiment of the present disclosure.

[0102] As shown in FIG3B , the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, and the first target resolution may be equal to the preset resolution, both being 2048×2048. Comparison shows that the original resolution of the first original image 301 is smaller than the preset resolution. Therefore, the first original image 301 may be scaled to obtain a third original image 305 having a resolution of 2048×1024. Based on the target aspect ratio, the image may be expanded based on the upper and lower boundaries of the third original image 305, centered on the third original image 305, to obtain a second expanded image 306 having a resolution of 2048×2048. The second extended image 306 may include a first sub-mask image 3041 adjacent to the upper boundary of the third original image 305 and a second sub-mask image 3042 adjacent to the lower boundary of the third original image 305. The first sub-mask image 3041 and the second sub-mask image 3042 may constitute the first mask image 304. Since the first target resolution is equal to the preset resolution, the third original image 305 is the second original image 303.

[0103] According to an embodiment of the present disclosure, the first target resolution may generally be greater than the preset resolution. If the first target resolution is not equal to the preset resolution, the first extended image may be scaled based on the first target resolution and the preset resolution to obtain a third extended image; and the second original image and the first mask image may be determined from the third extended image.

[0104] According to an embodiment of the present disclosure, the scaling process for the first extended image may be performed using the same or similar method as that for the scaling process for the first original image. For the description of the scaling process for the first extended image, specific reference is made to the description of the scaling process for the first original image, which will not be repeated here.

[0105] FIG3C schematically shows a schematic diagram of a process of generating a second original image and a first mask image according to yet another embodiment of the present disclosure.

[0106] As shown in Figure 3C , the original resolution of the first original image 301 may be 1024×512, the target aspect ratio may be 1:1, the first target resolution may be 2048×2048, and the preset resolution may indicate a long-side resolution of 512. By comparison, it is readily apparent that the long-side resolution in the original resolution is greater than the long-side resolution indicated by the preset resolution, and the first target resolution is greater than the original resolution. Therefore, the first original image 301 may be scaled based on the preset resolution to obtain a third original image 305 with a resolution of 512×256. Based on the target aspect ratio, the image may be expanded based on the upper boundary of the third original image 305, centered around the third original image 305, to obtain a second expanded image 306 with a resolution of 512×512. The second expanded image 306 may include the third original image 305 and an expanded portion 307. Because the first target resolution and the preset resolution are not equal, super-resolution upscaling can be performed on the second extended image 306 based on the first target resolution to obtain a third extended image 308 with a resolution of 2048×2048. After super-resolution upscaling, the third original image 305 can be upscaled to a second original image 303 with a resolution of 2048×1024, and the extended portion 307 can be upscaled to a first mask image 304 with a resolution of 2048×1024. Thus, the second original image 303 and the first mask image 304 can be determined from the third extended image 308.

[0107] According to an embodiment of the present disclosure, the second original image and the first mask image are obtained by reducing, expanding, and enlarging the image. When the first target resolution is large, the size of the image during the image expansion operation can be effectively reduced, thereby reducing the consumption of computing resources and improving computing efficiency.

[0108] According to an embodiment of the present disclosure, after the second original image and the first mask image are generated, a diffusion model may be used to generate a target extended image.

[0109] FIG4 schematically shows a flow chart of generating a target extended image.

[0110] As shown in FIG. 4 , the diffusion model may include a multimodal encoder 401 , an image information creator 402 , and a model decoder 403 .

[0111] According to an embodiment of the present disclosure, the second original image 404 and the image description text 405 may be encoded using a multimodal encoder 401 to obtain image encoding features 406 and text encoding features 407 .

[0112] According to an embodiment of the present disclosure, the multimodal encoder 401 can be used to encode the second original image 404 and the image description text 405, respectively. Alternatively, the multimodal encoder 401 can be divided into an image encoder 4011 and a text encoder 4012. The image encoder 4011 can encode the second original image 404 to obtain image encoding features 406. The text encoder 4012 can encode the image description text 405 to obtain text encoding features 407. The image encoder 4011 can be implemented, for example, using a variational autoencoder (VAE), and the text encoder 4012 can be implemented, for example, using the text encoder in the CLIP model.

[0113] According to an embodiment of the present disclosure, before processing the first mask image 408 using the diffusion model, the first mask image 408 may be initialized using Gaussian noise to obtain a noise image 409. The image data of the noise image 409 may obey a normal distribution.

[0114] According to an embodiment of the present disclosure, the image information creator 402 may be used to process the noise image 409 based on the image encoding feature 406 and the text encoding feature 407 to obtain the target output feature 410 .

[0115] According to an embodiment of the present disclosure, the target output features 410 can be decoded using the model decoder 403 to obtain a first target mask image 411. The model decoder 403 can correspond to the multimodal encoder 401, or the model decoder 403 can correspond to the image encoder 4011 in the multimodal encoder 401. Through the model decoder 403, the image features can be mapped back to the pixel space to obtain the first target mask image 411.

[0116] According to an embodiment of the present disclosure, the second original image 404 and the first target mask image 411 can be spliced ​​based on at least one edge of the second original image 404 used as the starting point for image expansion to obtain a target extended image 412 to complete the image generation process.

[0117] According to an embodiment of the present disclosure, the image information creator 402 may be a U-Net neural network structure for predicting noise, that is, the image information creator 402 may be composed of an encoding module, a decoding module, and an intermediate module. The encoding module may include M encoding functional blocks, and correspondingly, the decoding module may include M decoding functional blocks.

[0118] According to an embodiment of the present disclosure, using image coding features and text coding features as semantic benchmarks, processing a noisy image using an image information creator to obtain target output features may include the following operations:

[0119] The encoding module is used to process the image encoding features, the text encoding features and the noise image to obtain the first noise image features output by each of the M encoding function blocks included in the encoding module, where M is a positive integer; the intermediate module is used to process the first noise image features output by the Mth encoding function block to obtain the second noise image features; and the decoding module is used to process the second noise image features, the first noise image features output by each of the M encoding function blocks, the image encoding features and the text encoding features to obtain the target output features.

[0120] The following takes M=4 as an example, that is, the encoding module includes 4 encoding functional blocks and the decoding module includes 4 decoding functional blocks, to illustrate the processing flow of the image information creator 402.

[0121] FIG5 schematically shows a structural diagram of an image information creator according to an embodiment of the present disclosure.

[0122] As shown in FIG5 , the image information creator 402 may include multiple functional blocks from its input end to its output end, namely, four encoding functional blocks 4021 , one intermediate module 4022 and four decoding functional blocks 4023 .

[0123] According to an embodiment of the present disclosure, the four encoding function blocks 4021 and the four decoding function blocks 4023 included in the image information creator 402 can be divided into four levels. The encoding function blocks 4021 at each level can be jump-connected to the decoding function blocks 4023. The first noise image features output by the encoding function blocks 4021 can be provided as input to the decoding function blocks 4023 at the same level. The first encoding function block 4021 can be located at the same level as the fourth decoding function block 4023, the second encoding function block 4021 can be located at the same level as the third decoding function block 4023, the third encoding function block 4021 can be located at the same level as the second decoding function block 4023, and the fourth encoding function block 4021 can be located at the same level as the first decoding function block 4023.

[0124] According to an embodiment of the present disclosure, the M encoding function blocks 4021 may be connected via a cross attention block 4024. Each cross attention block 4024 may process the input features of the cross attention block 4024 based on the image encoding features 406 and the text encoding features 407.

[0125] Specifically, using the encoding module to process the image encoding feature, the text encoding feature, and the noise image to obtain the first noise image feature output by each of the M encoding function blocks included in the encoding module may include the following operations:

[0126] Use the j-th encoding function block to process the input features of the j-th encoding function block to obtain the first noise image features output by the j-th encoding function block, wherein j is a positive integer less than or equal to M, and the input features of the 1st encoding function block are represented as noise images; and when j is less than M, use the cross-attention block between the j-th encoding function block and the j+1-th encoding function block to process the first noise image features, image coding features, and text coding features output by the j-th encoding function block to obtain the input features of the j+1-th encoding function block.

[0127] According to an embodiment of the present disclosure, specifically, the input feature of the first encoding function block 4021 can be the image data of the noise image 409. The first encoding function block 4021 processes the image data of the noise image 409 to obtain the first noise image feature output by the first encoding function block 4021. The cross attention block 4024 located between the first encoding function block 4021 and the second encoding function block 4021 can process the first noise image feature output by the first encoding function block 4021 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the second encoding function block 4021.

[0128] The second encoding function block 4021 may process the input features of the second encoding function block 4021 to obtain a first noise image feature output by the second encoding function block 4021. The cross attention block 4024 located between the second encoding function block 4021 and the third encoding function block 4021 may process the first noise image feature output by the second encoding function block 4021 based on the image encoding features 406 and the text encoding features 407 to obtain an input feature for the third encoding function block 4021.

[0129] The third encoding function block 4021 may process the input features of the third encoding function block 4021 to obtain a first noise image feature output by the third encoding function block 4021. The cross attention block 4024 located between the third encoding function block 4021 and the fourth encoding function block 4021 may process the first noise image feature output by the third encoding function block 4021 based on the image encoding features 406 and the text encoding features 407 to obtain an input feature for the fourth encoding function block 4021.

[0130] The fourth encoding function block 4021 may process the input feature of the fourth encoding function block 4021 to obtain the first noise image feature output by the fourth encoding function block 4021 .

[0131] According to an embodiment of the present disclosure, the intermediate module 4022 may process the first noise image feature output by the fourth encoding function block 4021 to obtain a second noise image feature.

[0132] According to an embodiment of the present disclosure, the M decoding function blocks 4023 may be connected via a cross attention block 4024. Each cross attention block 4024 may process the input features of the cross attention block 4024 based on the image encoding features 406 and the text encoding features 407.

[0133] According to an embodiment of the present disclosure, using a decoding module to process the second noise image feature, the first noise image feature, the image coding feature, and the text coding feature output by each of the M coding function blocks to obtain a target output feature may include the following operations:

[0134] The kth decoding functional block is used to process the input features of the kth decoding functional block and the first noise image features output by the kth encoding functional block to obtain the decoding features output by the kth decoding functional block, wherein k is a positive integer less than or equal to M, and the input features of the first decoding functional block are represented as the second noise image features; when k is less than M, the cross-attention block between the kth decoding functional block and the k+1th decoding functional block is used to process the decoding features, image encoding features and text encoding features output by the kth decoding functional block to obtain the input features of the k+1th decoding functional block; and when k is equal to M, the target output feature is determined to be the decoding feature output by the kth decoding functional block.

[0135] According to an embodiment of the present disclosure, specifically, the input feature of the first decoding function block 4023 can be a third image feature. The first decoding function block 4023 can process the third image feature and the first noise image feature output by the fourth encoding function block 4021 obtained via a skip connection to obtain the decoded feature output by the first decoding function block 4023. The cross attention block 4024 located between the first decoding function block 4023 and the second decoding function block 4023 can process the decoded feature output by the first decoding function block 4023 based on the image encoding feature 406 and the text encoding feature 407 to obtain the input feature of the second decoding function block 4023.

[0136] The second decoding function block 4023 can process the input features of the second decoding function block 4023 and the first noise image features output by the third encoding function block 4021 obtained via the skip connection to obtain the decoded features output by the second decoding function block 4023. The cross attention block 4024 located between the second decoding function block 4023 and the third decoding function block 4023 can process the decoded features output by the second decoding function block 4023 based on the image encoding features 406 and the text encoding features 407 to obtain the input features of the third decoding function block 4023.

[0137] The third decoding function block 4023 can process the input features of the third decoding function block 4023 and the first noise image features output by the second encoding function block 4021 obtained via the skip connection to obtain the decoded features output by the third decoding function block 4023. The cross attention block 4024 located between the third decoding function block 4023 and the fourth decoding function block 4023 can process the decoded features output by the third decoding function block 4023 based on the image encoding features 406 and the text encoding features 407 to obtain the input features of the fourth decoding function block 4023.

[0138] The fourth decoding function block 4023 can process the input features of the fourth decoding function block 4023 and the first noise image features output by the first encoding function block 4021 obtained via the jump connection to obtain the decoding features output by the fourth decoding function block 4023, that is, the target output features 410.

[0139] According to an embodiment of the present disclosure, the encoding function block 4021 , the intermediate module 4022 and the decoding function block 4023 included in the image information creator 402 may all be implemented using a residual block (ResBlock).

[0140] The following describes the processing flow of the residual block by taking the process in which the intermediate module 4022 processes the first noise image feature output by the M-th encoding function block 4021 to obtain the second noise image feature as an example.

[0141] FIG6A schematically shows a structural diagram of an intermediate module according to an embodiment of the present disclosure.

[0142] As shown in FIG6A , the intermediate module 4022 can be implemented using a residual block. The first noise image feature output by the Mth encoding function block 4021 input to the intermediate module 4022 can be subjected to normalization, convolution, normalization, masking (dropout), and convolution in sequence to obtain a residual output feature. The residual output feature can be added to the convolved first noise image feature output by the Mth encoding function block 4021 to obtain a second noise image feature.

[0143] According to an embodiment of the present disclosure, as an optional implementation, a gated convolution block (GnBlock) can be used to replace the residual module, that is, the encoding function block 4021, the intermediate module 4022 and the decoding function block 4023 can all be implemented using a gated convolution block.

[0144] The following describes the processing flow of the gated convolution block by taking the process in which the intermediate module 4022 processes the first noise image feature output by the M-th encoding function block 4021 to obtain the second noise image feature as an example.

[0145] FIG6B schematically shows a structural diagram of an intermediate module according to another embodiment of the present disclosure.

[0146] As shown in FIG6B , the intermediate module 4022 can be implemented using a gated convolution block. For the first noise image feature output by the Mth encoding function block 4021 input to the intermediate module 4022, the first noise image feature output by the Mth encoding function block can be subjected to layer normalization to obtain a first sub-feature; the first sub-feature can be processed using a gated convolution layer (GnConv) to obtain a second sub-feature; the second sub-feature can be subjected to layer normalization to obtain a third sub-feature; the third sub-feature can be processed using a perceptron to obtain a fourth sub-feature; and a second noise image feature can be obtained based on the first noise image feature and the fourth sub-feature output by the Mth encoding function block.

[0147] According to an embodiment of the present disclosure, layer normalization processing may refer to normalizing the inputs of all neurons in batches, that is, making the data within the layer obey a normal distribution with a mean of 0 and a variance of 1, thereby speeding up processing efficiency and improving robustness to scaling changes during image processing.

[0148] According to an embodiment of the present disclosure, a perceptron may include multiple fully connected layers. Optionally, the multiple fully connected layers may be masked (dropout), i.e., random positions and numbers of neurons in the multiple fully connected layers are controlled to not participate in the calculation of the output result.

[0149] FIG6C schematically shows a structural diagram of a gated convolutional layer according to another embodiment of the present disclosure.

[0150] As shown in Figure 6C, for the first sub-feature input into the gated convolution layer, the first sub-feature can be convolved to obtain the fifth sub-feature. The fifth sub-feature can be subjected to deep convolution (Deep-Wise Conv) to obtain a deep convolution feature. The gated convolution layer can include P dual-input gated processing units, where P can be a positive integer, one input of the gated processing unit can be a deep convolution feature, and the other input can be the output feature of the previous gated processing unit. Each gated processing unit can perform a dot product on its two inputs and perform a convolution on the product to obtain its output feature. For example, for the p-th gated processing unit, the p-th gated processing unit can perform a dot product on the deep convolution feature and the gated output feature output by the p-1-th gated processing unit to obtain the sixth sub-feature of the p-th gated processing unit, and then perform a convolution on the sixth sub-feature of the p-th gated processing unit to obtain the gated output feature of the p-th gated processing unit. Wherein, p can be expressed as a positive integer, the gated output feature output by the 0th gated processing unit can be expressed as the fifth sub-feature, and the gated output feature output by the Pth gated processing unit can be expressed as the second sub-feature.

[0151] According to an embodiment of the present disclosure, taking three gated processing units as an example, the depthwise convolution feature can be dot-multiplied with the fifth sub-feature, and the product can be convolved to obtain the sixth sub-feature. The depthwise convolution feature can be dot-multiplied with the sixth sub-feature, and the product can be convolved to obtain the seventh sub-feature. The depthwise convolution feature can be dot-multiplied with the seventh sub-feature, and the product can be convolved to obtain the second sub-feature of the output.

[0152] According to an embodiment of the present disclosure, the gated convolution layer in the gated convolution block can use gated convolution and recursion to achieve interaction between different spatial features, enhance feature expression, and thus improve the generation effect of the diffusion model.

[0153] According to embodiments of the present disclosure, in some application scenarios, the first target resolution may differ significantly from the original resolution. When this difference is significant, the content of the image portion obtained through image expansion may differ significantly from the first original image, and the content and style of the expanded image portion may be inconsistent with the first original image. Therefore, as an optional implementation, if it is determined that a preset condition is met based on the first target resolution and the original resolution of the first original image, the image expansion may be performed N times using the first original image and the image description text as semantic references to obtain the target expanded image, where N is a positive integer.

[0154] According to an embodiment of the present disclosure, the preset condition may be that the ratio between the first target resolution and the original resolution of the first original image is less than a first threshold; or, the ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold.

[0155] According to an embodiment of the present disclosure, the ratio between the first target resolution and the original resolution of the first original image can be expressed as the ratio between the long side resolution in the first target resolution and the long side resolution in the original resolution, or as the ratio between the short side resolution in the first target resolution and the short side resolution in the original resolution, or as the smaller of the above two ratios, which is not limited here.

[0156] According to an embodiment of the present disclosure, the first threshold and the second threshold can be set according to a specific application scenario. For example, the first threshold can be set to 0.66, the second threshold can be set to 1.5, etc., which is not limited here.

[0157] According to an embodiment of the present disclosure, the size of N may be determined based on the first target resolution and the original resolution. For example, the ratio between the first target resolution and the original resolution may be 2.25, and the resolution of each expansion may not exceed 50% of the image resolution. The target expanded image may be obtained by performing two image expansions on the first original image.

[0158] According to embodiments of the present disclosure, users can also configure the number of image extensions. Accordingly, the image extension request can include information about the number of image extensions. As an optional implementation, the preset condition can also include the number of image extensions included in the image extension request being greater than 1. Accordingly, in this case, N represents the number of image extensions.

[0159] According to an embodiment of the present disclosure, performing N image expansions using the first original image and the image description text as semantic references to obtain a target expanded image may include the following operations:

[0160] When the first target resolution and the image resolution of the extended image after the i-1th image expansion are not equal, a second target resolution is determined based on the image resolution and expansion ratio of the extended image after the i-1th image expansion, where i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution; using the target aspect ratio and the second target resolution, based on the extended image after the i-1th image expansion, a fourth original image and a second mask image are generated; using the image description text and the fourth original image as semantic references, image generation is performed based on the second mask image to obtain a second target mask image; and the fourth original image and the second target mask image are image stitched to obtain the extended image after the i-1th image expansion.

[0161] According to an embodiment of the present disclosure, when the first target resolution and the image resolution of the expanded image after the i-th image expansion are equal, the expanded image after the i-th image expansion can be determined as the target expanded image. Alternatively, when the value of i is equal to N, the expanded image after the i-th image expansion can be determined as the target expanded image.

[0162] FIG7 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.

[0163] As shown in FIG. 7 , the method includes operations S701 to S708 .

[0164] In operation S701 , an expansion magnification is determined based on a first target resolution, an original resolution of a first original image, and N, and i=0 is initialized.

[0165] In operation S702 , the value of i is controlled to be incremented by 1.

[0166] In operation S703 , it is determined whether i is less than N. If it is determined that i is less than N, operation S704 is performed. If it is determined that i is greater than or equal to N, operation S708 is performed.

[0167] In operation S704, a second target resolution is determined based on the image resolution and the expansion magnification of the expanded image after the (i-1)th image expansion.

[0168] In operation S705 , a fourth original image and a second mask image are generated based on the expanded image after the (i-1)th image expansion using the target aspect ratio and the second target resolution.

[0169] In operation S706 , an image is generated based on the second mask image using the image description text and the fourth original image as semantic references to obtain a second target mask image.

[0170] In operation S707, the fourth original image is spliced ​​with the second target mask image to obtain an expanded image after the i-th image expansion. After operation S707 is completed, the process returns to operation S702.

[0171] In operation S708 , the extended image after the i-th image expansion is determined as a target extended image.

[0172] According to an embodiment of the present disclosure, the image expansion effect will be described below with reference to FIG. 8A to FIG. 8C , assuming that the number of image expansions is 2 and taking the first original image shown in FIG. 8A as an example.

[0173] FIG8A schematically shows a schematic diagram of a first original image according to another embodiment of the present disclosure.

[0174] As shown in FIG8A , the original resolution of the first original image may be 512×512, the first target resolution may be 2560×1280, and the target aspect ratio may be 2:1. Therefore, the ratio between the first target resolution and the original resolution may be determined to be 2.5, and the resolution may be expanded by 1.5 times the original image resolution each time, i.e., the expansion ratio may be 1.5.

[0175] According to an embodiment of the present disclosure, by performing visual language recognition on the first original image, labels such as people, coins, blackboards, and charts can be obtained, and image description text can be generated based on the above labels.

[0176] According to an embodiment of the present disclosure, the second target resolution during the first expansion can be determined to be 768×768 based on the original resolution and the expansion magnification, and the fourth original image can be obtained accordingly.

[0177] According to an embodiment of the present disclosure, based on the target aspect ratio, a second mask image with a resolution of 768×768 may be added to the fourth original image.

[0178] According to an embodiment of the present disclosure, the second mask image may be processed using a diffusion model based on the image description text and the fourth original image to obtain an image after a single expansion.

[0179] FIG8B schematically shows a schematic diagram of an image after one expansion according to another embodiment of the present disclosure.

[0180] As shown in FIG8B , the resolution of the image after one expansion may be 1536×768.

[0181] According to an embodiment of the present disclosure, based on the resolution and expansion ratio of the image after the first expansion, the second target resolution for the second expansion is determined to be 2304×1152, and the fourth original image can be obtained accordingly.

[0182] According to an embodiment of the present disclosure, two second mask images with resolutions of 2304×128 and 256×1280 may be added to the fourth original image based on the target aspect ratio.

[0183] According to an embodiment of the present disclosure, the two second mask images may be processed using a diffusion model based on the image description text and the fourth original image to obtain a target expanded image.

[0184] FIG8C schematically shows a schematic diagram of a target extended image according to another embodiment of the present disclosure.

[0185] As shown in FIG8C , the resolution of the target expanded image may be 2560×1280.

[0186] According to an embodiment of the present disclosure, by performing image expansion multiple times, the quality of the generated image can be effectively improved, the content of the target expanded image can be kept consistent with the style of the first original image, and the display effect of the target expanded image can be improved.

[0187] FIG9 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.

[0188] As shown in FIG. 9 , the method 900 may include operations S910 to S960 .

[0189] In operation S910 , in response to a first task start control being triggered, a first original image is acquired from a return result of a first image input control included in a display interface.

[0190] In operation S920 , visual language recognition is performed on the first original image to obtain image description text.

[0191] In operation S930, a target aspect ratio and a first target resolution are obtained based on the return results of each of the plurality of first configuration controls included in the display interface.

[0192] In operation S940, a second original image and a first mask image are generated based on the first original image using the target aspect ratio and the first target resolution.

[0193] In operation S950 , an image is generated based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image.

[0194] In operation S960 , the first target mask image and the second original image are spliced ​​to obtain a target extended image.

[0195] According to an embodiment of the present disclosure, a display interface may refer to a display interface of a client application process running in an electronic device on a display screen of the electronic device. The electronic device may be any device that supports human-computer interaction.

[0196] According to an embodiment of the present disclosure, the first task start control may be a control for indicating the start of an image generation task, and the first task start control may be represented as a key or a button on a display interface. A user may touch and click the corresponding key or button, or the user may control a mouse to click the corresponding key or button to trigger the first task start control. When the first task start control is triggered, the electronic device may generate an image expansion request so that a client application process within the electronic device responds to the image expansion request to expand the image.

[0197] According to an embodiment of the present disclosure, the first image input control can be represented as a larger blank box on the real interface. The user can input the first original image based on the image expansion task into the blank box by dragging, loading, etc. The first image input control can use the first original image as return information, which can facilitate the process in the electronic device to obtain the first original image by accessing the first image input control.

[0198] According to an embodiment of the present disclosure, the plurality of first configuration controls may include an input box type configuration control, an option type configuration control, a slider type configuration control, and the like. Depending on the type of configuration control, a user may input information through different operations. The information received by the configuration control may be used as return information, facilitating a process within the electronic device to obtain the corresponding information by accessing the configuration control.

[0199] In another embodiment of the present disclosure, a method of expanding the first original image to obtain a target expanded image can be implemented using the image processing method of the embodiment of the present disclosure, which will not be described in detail here.

[0200] The method shown in FIG. 9 will be further described below with reference to FIG. 10A to FIG. 10C in combination with specific embodiments.

[0201] According to an embodiment of the present disclosure, the plurality of first configuration controls may include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control. The horizontal resolution configuration control, the vertical resolution configuration control, the horizontal expansion ratio configuration control, and the vertical resolution configuration control may all be configuration controls of the input box type.

[0202] According to an embodiment of the present disclosure, obtaining a target aspect ratio and a first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface may include the following operations:

[0203] Determine the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control; and determine the target aspect ratio based on the first target resolution.

[0204] According to an embodiment of the present disclosure, the horizontal expansion ratio information may refer to the ratio of the horizontal size of the extended image obtained after horizontal expansion based on the first original image to the horizontal size of the first original image. Correspondingly, the vertical expansion ratio information may refer to the ratio of the vertical size of the extended image obtained after vertical expansion based on the first original image to the vertical size of the first original image. Taking the multi-screen expansion scenario as an example, the first original image can be matched with a single screen, the horizontal expansion ratio information can be expressed as the number of horizontal screens, and the vertical expansion ratio information can be expressed as the number of vertical screens. For example, if the original single screen needs to be expanded to 2 rows and 3 columns with a total of 4 screens, the horizontal expansion ratio information can be set to 2 and the vertical expansion ratio information can be set to 3.

[0205] According to an embodiment of the present disclosure, the horizontal resolution information and the vertical resolution information may be expressed as target resolutions for adjusting the first original image when horizontal and vertical expansion is not performed, that is, target resolutions when both the horizontal expansion ratio information and the vertical expansion ratio information are expressed as 1. In a multi-screen expansion scenario, the horizontal resolution information and the vertical resolution information may be expressed as the resolution of the image on a single screen.

[0206] According to an embodiment of the present disclosure, the horizontal resolution in the first target resolution can be obtained by multiplying the horizontal resolution represented by the horizontal resolution information by the horizontal expansion ratio represented by the horizontal expansion ratio information. Correspondingly, the vertical resolution in the first target resolution can be obtained by multiplying the vertical resolution represented by the vertical resolution information by the vertical expansion ratio represented by the vertical expansion ratio information.

[0207] According to an embodiment of the present disclosure, the horizontal resolution in the first target resolution may be ratioed to the vertical resolution in the first target resolution to obtain a target aspect ratio.

[0208] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include an expansion direction configuration control.

[0209] According to an embodiment of the present disclosure, generating a second original image and a first mask image based on a first original image using a target aspect ratio and a first target resolution may include the following operations:

[0210] Based on the expansion direction information returned by the expansion direction configuration control, at least one target edge is determined from the four edges of the first original image; based on the target aspect ratio and the first target resolution, image expansion is performed with the at least one target edge of the first original image as a starting point to obtain a first extended image; and a second original image and a first mask image are determined from the first extended image, wherein the second original image is the first original image.

[0211] According to an embodiment of the present disclosure, the extended direction configuration control can be a selection-type configuration control. For example, the extended direction configuration control can include four selection items representing "up," "down," "left," and "right," respectively. The user can select one or more of these four selection items to determine at least one target edge from the four edges of the first original image. Specifically, the user can select the "up" and "down" selection items, and the target edge can include the upper and lower edges of the first original image.

[0212] According to an embodiment of the present disclosure, multiple first configuration controls may further include a candidate image quantity control. The candidate image quantity control may be an input box type configuration control or a slider type configuration control. The user may enter a number in the input box of the candidate image quantity control, or slide the reference cursor to the left or right at the slider of the candidate image quantity control to determine the candidate image quantity information. The candidate image quantity information may be represented as the number of canvas images generated before the image expansion is generated. Specifically, depending on the expansion direction of the selected image, when the expansion direction includes two relative directions, such as including both the "up" and "down" directions, or including both the "left" and "right" directions, the pixels of the canvas expanded in the two directions may be different, thereby generating multiple selectable canvas images. The candidate image quantity information is represented as the number of generated canvas images.

[0213] According to an embodiment of the present disclosure, based on a target aspect ratio and a first target resolution, image expansion is performed with at least one target edge of the first original image as a starting point to obtain a first extended image, which may include the following operations:

[0214] Determine the number of candidate images based on the candidate image quantity information returned by the candidate image quantity control; based on the target aspect ratio and the first target resolution, perform image expansion with at least one target edge of the first original image as the starting point to obtain a fifth extended image of the number of candidate images; send the fifth extended image of the number of candidate images to the first image output control, wherein the first image output control is configured to display the fifth extended image of the number of candidate images on the display interface; and determine the first extended image from the fifth extended image of the number of candidate images in response to a selection request for the fifth extended image of the number of candidate images.

[0215] According to an embodiment of the present disclosure, the fifth extended image is the expanded canvas image. The fifth extended image may include the original first original image and extended partial images with different shapes, colors, etc.

[0216] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a generation speed configuration control. The generation speed configuration control can be an input box type configuration control or a slider type configuration control. A user can enter a number in the input box of the generation speed configuration control, or slide a reference cursor left or right on the slider of the generation speed configuration control to determine the generation speed information.

[0217] According to an embodiment of the present disclosure, the generation speed information can be represented as the number of rounds of image expansion, i.e., the number of image expansions. The smaller the value represented by the generation speed information, the slower the image generation speed, and accordingly, the greater the number of image expansions determined by the generation speed information.

[0218] According to an embodiment of the present disclosure, multiple first configuration controls may further include a single maximum expansion pixel control, which may be an input box type configuration control or a slider type configuration control. The user may enter a number in the input box of the single maximum expansion pixel control, or slide the reference cursor to the left or right at the slider of the single maximum expansion pixel control to determine the single maximum expansion pixel information. The single maximum expansion pixel information may be expressed as the maximum allowable horizontal pixel expansion amount and vertical pixel expansion amount for a single image expansion. For example, if the single maximum expansion pixel information is expressed as 256, it means that when a single image is expanded, the horizontal pixels are expanded or reduced by a maximum of 256 pixels, and the vertical pixels are expanded or reduced by a maximum of 256 pixels. When the difference between the first target pixel and the original pixel of the first original image is greater than 256 pixels, the image expansion may be performed multiple times, that is, the number of image expansions is determined to be greater than 1.

[0219] According to an embodiment of the present disclosure, the number of image expansions can be determined based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image, and the first target resolution; and when the number of image expansions is greater than 1, based on the number of image expansions, multiple image expansions are performed with the first original image and the image description text as semantic benchmarks to obtain a target expanded image.

[0220] According to an embodiment of the present disclosure, specifically, the first image expansion times can be determined based on the generation speed information returned by the generation speed configuration control; the second image expansion times can be determined based on the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image and the first target resolution; and the maximum value of the first image expansion times and the second image expansion times is taken as the determined image expansion times.

[0221] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include an image style configuration control. The image style configuration control may be a selection item type configuration control, which may include multiple selection items representing image styles, such as a selection item representing an image style of "poster," a selection item representing an image style of "landscape," and a selection item representing an image style of "architecture." The user may determine the image style information by selecting an option.

[0222] According to an embodiment of the present disclosure, performing visual language recognition on the first original image to obtain image description text may include the following operations:

[0223] Based on the image style information returned by the image style configuration control, image extension keywords are obtained; the first original image is processed using a visual language pre-training model to obtain at least one object text; and image description text is obtained based on the at least one object text and the image extension keywords.

[0224] According to an embodiment of the present disclosure, the plurality of first configuration controls may further include a creativity configuration control. The creativity configuration control may be an input box type configuration control or a slider type configuration control. A user may enter a number in the input box of the creativity configuration control, or slide a reference cursor left or right on the slider of the creativity configuration control to determine the creativity information.

[0225] According to an embodiment of the present disclosure, the creativity information may be represented by the similarity between the generated target extended image and the original first original image. The higher the creativity represented by the creativity information, the lower the similarity between the generated target extended image and the original first original image.

[0226] According to embodiments of the present disclosure, the influence of the image coding features and text coding features on the generation process of the target extended image can be modified by weighting the image coding features and text coding features, thereby controlling the similarity between the generated target extended image and the original first original image. The higher the creativity indicated by the creativity information, the smaller the weights of the determined image coding features and text coding features can be.

[0227] According to an embodiment of the present disclosure, specifically, using the image description text and the second original image as semantic references, performing image generation based on the first mask image to obtain the first target mask image may include the following operations:

[0228] The second original image and the image description text are respectively encoded using a multimodal encoder to obtain initial image coding features and initial text coding features; based on the creativity information returned by the creativity configuration control, the initial image coding features and the initial text coding features are corrected to obtain image coding features and text coding features; the first mask image is initialized using Gaussian noise to obtain a noise image; the image coding features and the text coding features are used as semantic benchmarks, and the noise image is processed using an image information creator to obtain target output features; and the target output features are decoded using a model decoder to obtain a first target mask image.

[0229] According to an embodiment of the present disclosure, the current weight of each of the initial image coding feature and the initial text coding feature may be 1. A weight with a value of 0 to 1 may be determined based on the creativity information, and the initial image coding feature and the initial text coding feature may be modified using the weight. Specifically, a vector multiplication operation may be performed on the weight with the initial image coding feature and the initial text coding feature to obtain the image coding feature and the text coding feature.

[0230] FIG10A schematically shows a schematic diagram of a first display interface according to an embodiment of the present disclosure.

[0231] As shown in FIG10A , the first display interface may represent a portion of the display interface before image expansion. This display interface may include a first task start control, a first image input control, and configuration controls such as a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, a vertical resolution configuration control, an expansion direction configuration control, a candidate image quantity control, a generation speed configuration control, a single maximum expansion pixel control, an image style configuration control, and a creativity configuration control. Before image expansion, the user may place the first original image into the first image input control and complete information configuration in each configuration control. Specifically, the configuration information filled in by the user may include setting the generation speed information to 2, the maximum single expansion pixel to 256, the image style information to "poster", the creativity information to 2, the expansion direction information to "left" and "right", that is, expanding to the left and right sides based on the first original image, the number of candidate images information is set to 4, that is, 4 candidate fifth extended images will be generated for the user to select, the horizontal resolution information is set to 1024, the vertical resolution information is set to 768, the horizontal expansion ratio information is set to 1, and the vertical expansion ratio information is set to 1, which can determine that the first target resolution is 1024×768 and the target aspect ratio is 4:3.

[0232] According to an embodiment of the present disclosure, after determining that the first original image has been input and configuration has been completed, the user can click the first task launch control to initiate the image expansion task. Alternatively, if determining that the first original image has been input but configuration has not been completed, if the user clicks the first task launch control, the corresponding configuration controls can be filled in using the preset values ​​or initial values ​​corresponding to the respective unconfigured configuration controls before initiating the image expansion task.

[0233] According to an embodiment of the present disclosure, in addition to the above-mentioned controls, the first display interface may also include other functional controls. For example, the first display interface may also include a resolution display control, which can be used to display the original resolution of the first original image. Specifically, after the user inputs the first original image into the first image input control, an image input request may be generated. In response to the image input request related to the first original image, the resolution information of the first original image may be determined; and the resolution information may be sent to the resolution display control, wherein the resolution display control is configured to display the original resolution of the first original image on the display interface based on the resolution information. As shown in Figure 10A, after the first original image is input into the first image input control, the original resolution of the first original image may be displayed as 640×640.

[0234] According to an embodiment of the present disclosure, for another example, the first display interface may further include an initialization control, and the initialization control may be used to initialize configuration information of the plurality of first configuration controls.

[0235] According to an embodiment of the present disclosure, after it is determined that the first task start control is clicked, a fifth expanded image of the number of candidate images may be displayed on the display interface of the electronic device.

[0236] FIG10B schematically shows a schematic diagram of a second display interface according to an embodiment of the present disclosure.

[0237] As shown in Figure 10B, the second display interface can represent the portion of the display interface during the execution of the image expansion task, and the second display interface can be displayed side by side with the first display interface. The second display interface can include a first image output control, and the first image output control can include a selection box, within which the fifth expanded image can be displayed. Optionally, the selection box can also include thumbnails of each fifth expanded image, and the user can select the thumbnail to determine the fifth expanded image displayed in the selection box.

[0238] According to an embodiment of the present disclosure, the first image output control may further include multiple selection options, wherein the selection options for the number of first candidate images in the multiple selection options may correspond one-to-one with the number of fifth expanded images in the candidate image list. The first image output control may further include a display box, wherein the user may select a first expanded image by checking a box from the multiple selection options. The first expanded image displayed in the display box may be different from the fifth expanded image displayed in the selection box.

[0239] According to an embodiment of the present disclosure, the second display interface may optionally include a key or button-type control that can indicate that the first extended image has been selected and that generation of the target extended image has begun. The name of the control on the second display interface may be, for example, "Start Super-Resolution" as shown in FIG10B . The user can click the control to start generation of the target extended image.

[0240] FIG10C schematically shows a schematic diagram of a third display interface according to an embodiment of the present disclosure.

[0241] As shown in FIG10C , the third display interface may represent the portion of the display interface after the image expansion task is completed, and the third display interface may be displayed side by side with the first display interface. The third display interface may include a second image output control.

[0242] According to an embodiment of the present disclosure, the target expanded image may be sent to the second image output control, wherein the second image output control is configured to display the target expanded image on a display interface.

[0243] FIG11 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0244] As shown in FIG. 11 , the image processing apparatus 1100 includes a first recognition module 1110 , a first generation module 1120 , a first image generation module 1130 , and a first stitching module 1140 .

[0245] The first recognition module 1110 is configured to perform visual language recognition on the first original image in response to the image expansion request to obtain image description text.

[0246] The first generating module 1120 is configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution included in the image extension request.

[0247] The first image generation module 1130 is configured to generate an image based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image.

[0248] The first stitching module 1140 is configured to stitch the first target mask image with the second original image to obtain a target extended image.

[0249] According to an embodiment of the present disclosure, the first generation module 1120 includes a first generation submodule and a second generation submodule.

[0250] The first generating submodule is configured to perform image expansion based on a target aspect ratio and a first target resolution, taking at least one side of the first original image as a starting point, to obtain a first expanded image.

[0251] The second generating submodule is configured to determine a second original image and a first mask image from the first extended image, wherein the second original image is the first original image.

[0252] According to an embodiment of the present disclosure, the first generation module 1120 includes a third generation submodule, a fourth generation submodule, and a fifth generation submodule.

[0253] The third generating submodule is configured to perform scaling processing on the first original image based on the original resolution and the preset resolution of the first original image to obtain a third original image.

[0254] The fourth generating submodule is configured to perform image expansion based on the target aspect ratio and taking at least one side of the third original image as a starting point to obtain a second expanded image.

[0255] The fifth generating submodule is configured to determine the second original image and the first mask image from the second extended image when the first target resolution is equal to the preset resolution.

[0256] According to an embodiment of the present disclosure, the first generation module 1120 further includes a sixth generation submodule and a seventh generation submodule.

[0257] The sixth generating submodule is configured to, when the first target resolution is not equal to the preset resolution, perform scaling processing on the first extended image based on the first target resolution and the preset resolution to obtain a third extended image.

[0258] The seventh generating submodule is configured to determine a second original image and a first mask image from the third extended image.

[0259] According to an embodiment of the present disclosure, the image processing apparatus 1100 further includes a first expansion module.

[0260] The first expansion module is configured to, based on the first target resolution and the original resolution of the first original image, determine that a preset condition is met, and then perform N image expansions using the first original image and the image description text as semantic references to obtain a target expanded image, where N is a positive integer.

[0261] According to an embodiment of the present disclosure, the preset condition includes any one of the following: a ratio between the first target resolution and the original resolution of the first original image is less than a first threshold; a ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold; and an image extension request includes an image extension count greater than 1.

[0262] According to an embodiment of the present disclosure, the first expansion module includes a first expansion sub-module, a second expansion sub-module, a third expansion sub-module, a fourth expansion sub-module, and a fifth expansion sub-module.

[0263] The first expansion submodule is configured to determine an expansion ratio based on the first target resolution, the original resolution of the first original image, and N.

[0264] The second expansion submodule is used to determine the second target resolution based on the image resolution and expansion ratio of the extended image after the i-1th image expansion when the first target resolution and the image resolution of the extended image after the i-1th image expansion are not equal, wherein i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution.

[0265] The third expansion submodule is configured to generate a fourth original image and a second mask image based on the expanded image after the (i-1)th image expansion by using the target aspect ratio and the second target resolution.

[0266] The fourth expansion submodule is configured to use the image description text and the fourth original image as semantic references and perform image generation based on the second mask image to obtain a second target mask image.

[0267] The fifth expansion submodule is configured to perform image splicing on the fourth original image and the second target mask image to obtain an expanded image after the i-th image expansion.

[0268] According to an embodiment of the present disclosure, the first expansion module further includes a sixth expansion sub-module.

[0269] The sixth expansion submodule is configured to determine the expanded image after the i-th image expansion as the target expanded image when the first target resolution is equal to the image resolution of the expanded image after the i-th image expansion.

[0270] According to an embodiment of the present disclosure, the first image generating module 1130 includes a first image generating submodule, a second image generating submodule, a third image generating submodule, and a fourth image generating submodule.

[0271] The first image generation submodule is used to use a multimodal encoder to encode the second original image and the image description text respectively to obtain image encoding features and text encoding features.

[0272] The second image generation submodule is used to initialize the first mask image using Gaussian noise to obtain a noise image.

[0273] The third image generation submodule is used to use the image coding features and text coding features as semantic benchmarks, use the image information creator to process the noisy image, and obtain the target output features.

[0274] The fourth image generation submodule is used to decode the target output features using the model decoder to obtain a first target mask image.

[0275] According to an embodiment of the present disclosure, an image information creator includes an encoding module, a decoding module, and an intermediate module.

[0276] According to an embodiment of the present disclosure, the third image generation submodule includes a first image generation unit, a second image generation unit, and a third image generation unit.

[0277] The first image generating unit is used to process the image coding feature, the text coding feature and the noise image by using the coding module to obtain the first noise image feature output by each of the M coding function blocks included in the coding module, where M is a positive integer.

[0278] The second image generating unit is configured to process the first noise image feature output by the Mth encoding function block using the intermediate module to obtain a second noise image feature.

[0279] The third image generation unit is used to use the decoding module to process the second noise image feature, the first noise image feature output by each of the M encoding function blocks, the image coding feature and the text coding feature to obtain the target output feature.

[0280] According to an embodiment of the present disclosure, the M encoding functional blocks are connected through cross-attention blocks.

[0281] According to an embodiment of the present disclosure, the first image generating unit includes a first image generating sub-unit and a second image generating sub-unit.

[0282] The first image generation subunit is used to use the j-th encoding function block to process the input features of the j-th encoding function block to obtain the first noise image features output by the j-th encoding function block, wherein j is a positive integer less than or equal to M, and the input features of the first encoding function block are represented as a noise image.

[0283] The second image generation subunit is used to process the first noise image feature, image coding feature, and text coding feature output by the jth coding functional block using the cross-attention block between the jth coding functional block and the j+1th coding functional block when j is less than M, to obtain the input feature of the j+1th coding functional block.

[0284] According to an embodiment of the present disclosure, the decoding module includes M decoding functional blocks, and the M decoding functional blocks are connected by cross attention blocks.

[0285] According to an embodiment of the present disclosure, the third image generating unit includes a third image generating sub-unit, a fourth image generating sub-unit, and a fifth image generating sub-unit.

[0286] The third image generation subunit is used to use the kth decoding function block to process the input features of the kth decoding function block and the first noise image features output by the kth encoding function block to obtain the decoding features output by the kth decoding function block, wherein k is a positive integer less than or equal to M, and the input features of the first decoding function block are represented as the second noise image features.

[0287] The fourth image generation subunit is used to process the decoding features, image encoding features and text encoding features output by the kth decoding functional block using the cross-attention block between the kth decoding functional block and the k+1th decoding functional block when k is less than M, to obtain the input features of the k+1th decoding functional block.

[0288] The fifth image generation subunit is used to determine, when k is equal to M, that the target output feature is the decoding feature output by the kth decoding functional block.

[0289] According to an embodiment of the present disclosure, the intermediate module, the encoding function block and the decoding function block are gated convolution blocks.

[0290] According to an embodiment of the present disclosure, the second image generating unit includes a sixth image generating subunit, a seventh image generating subunit, an eighth image generating subunit, a ninth image generating subunit, and a tenth image generating subunit.

[0291] The sixth image generation subunit is configured to perform layer normalization processing on the first noise image feature output by the Mth encoding functional block to obtain a first sub-feature.

[0292] The seventh image generation subunit is configured to process the first sub-feature using a gated convolutional layer to obtain a second sub-feature.

[0293] The eighth image generation subunit is configured to perform layer normalization processing on the second sub-feature to obtain a third sub-feature.

[0294] The ninth image generation subunit is configured to process the third sub-feature using a perceptron to obtain a fourth sub-feature.

[0295] The tenth image generating subunit is configured to obtain a second noise image feature based on the first noise image feature and the fourth sub-feature output by the Mth encoding functional block.

[0296] According to an embodiment of the present disclosure, the gated convolution layer includes P gated image generation units, where P is a positive integer.

[0297] According to an embodiment of the present disclosure, the seventh image generation subunit includes a first image generation component, a second image generation component, a third image generation component, and a fourth image generation component.

[0298] The first image generation component is used to perform convolution processing on the first sub-feature to obtain a fifth sub-feature.

[0299] The second image generation component is used to perform depth convolution processing on the fifth sub-feature to obtain a depth convolution feature.

[0300] The third image generation component is used to perform a dot product process on the depth convolution feature and the gated output feature output by the p-1th gated image generation unit for the p-th gated image generation unit to obtain the sixth sub-feature of the p-th gated image generation unit, where p is a positive integer less than or equal to P.

[0301] a fourth image generation component, configured to perform convolution processing on the sixth sub-feature of the p-th gated image generation unit to obtain a gated output feature output by the p-th gated image generation unit, wherein the gated output feature of the 0th gated image generation unit is represented as the fifth sub-feature, and the gated output feature of the P-th gated image generation unit is represented as the second sub-feature.

[0302] According to an embodiment of the present disclosure, the first identification module 1110 includes a first identification submodule and a second identification submodule.

[0303] The first recognition submodule is used to process the first original image using a visual language pre-training model to obtain at least one object text.

[0304] The second recognition submodule is configured to obtain an image description text based on at least one object text.

[0305] According to an embodiment of the present disclosure, the image expansion request further includes an image expansion keyword.

[0306] According to an embodiment of the present disclosure, the first identification module 1110 includes a third identification submodule.

[0307] The third recognition submodule is configured to obtain an image description text based on at least one object text and an image expansion keyword.

[0308] It should be noted that the image processing device part in the embodiment of the present disclosure corresponds to the image processing method part in the embodiment of the present disclosure. The description of the image processing device part specifically refers to the image processing method part and will not be repeated here.

[0309] FIG12 schematically shows a block diagram of an image processing apparatus according to another embodiment of the present disclosure.

[0310] As shown in FIG. 12 , the image processing apparatus 1200 includes a first acquisition module 1210 , a second recognition module 1220 , a second acquisition module 1230 , a second generation module 1240 , a second image generation module 1250 and a second stitching module 1260 .

[0311] The first acquisition module 1210 is configured to acquire a first original image from a return result of a first image input control included in the display interface in response to the first task start control being triggered.

[0312] The second recognition module 1220 is configured to perform visual language recognition on the first original image to obtain image description text.

[0313] The second acquisition module 1230 is configured to obtain a target aspect ratio and a first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface.

[0314] The second generating module 1240 is configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution.

[0315] The second image generation module 1250 is configured to generate an image based on the first mask image using the image description text and the second original image as semantic references to obtain a first target mask image.

[0316] The second stitching module 1260 is configured to stitch the first target mask image with the second original image to obtain a target extended image.

[0317] According to an embodiment of the present disclosure, the plurality of first configuration controls include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control.

[0318] According to an embodiment of the present disclosure, the second acquisition module 1230 includes a first acquisition submodule and a second acquisition submodule.

[0319] The first acquisition submodule is used to determine the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control.

[0320] The second acquisition submodule is configured to determine a target aspect ratio based on the first target resolution.

[0321] According to an embodiment of the present disclosure, the plurality of first configuration controls further include an expansion direction configuration control.

[0322] According to an embodiment of the present disclosure, the second generation module 1240 includes an eighth generation submodule, a ninth generation submodule, and a tenth generation submodule.

[0323] The eighth generating submodule is configured to determine at least one target edge from the four edges of the first original image based on the expansion direction information returned by the expansion direction configuration control.

[0324] The ninth generating submodule is configured to perform image expansion based on the target aspect ratio and the first target resolution, taking at least one target edge of the first original image as a starting point, to obtain a first expanded image.

[0325] The tenth generating submodule is configured to determine a second original image and a first mask image from the first extended image, wherein the second original image is the first original image.

[0326] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a candidate image quantity control.

[0327] According to an embodiment of the present disclosure, the ninth generation submodule includes a first generation unit, a second generation unit, a third generation unit, and a fourth generation unit.

[0328] The first generating unit is configured to determine the number of candidate images based on the candidate image number information returned by the candidate image number control.

[0329] The second generating unit is configured to perform image expansion based on the target aspect ratio and the first target resolution, taking at least one target edge of the first original image as a starting point, to obtain fifth expanded images equal to the number of candidate images.

[0330] The third generating unit is configured to send a fifth expanded image of the number of candidate images to the first image output control, wherein the first image output control is configured to display the fifth expanded image of the number of candidate images on the display interface.

[0331] The fourth generating unit is configured to determine a first extended image from the fifth extended images of the candidate images in response to a selection request for the fifth extended image of the candidate images.

[0332] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a generation speed configuration control and a single maximum expansion pixel control.

[0333] According to an embodiment of the present disclosure, the image processing apparatus 1200 further includes a first determining module and a second expanding module.

[0334] The first determining module is configured to determine the number of image expansion times based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image, and the first target resolution.

[0335] The second expansion module is used to perform multiple image expansions based on the number of image expansions, taking the first original image and the image description text as semantic references, to obtain a target expanded image when the number of image expansions is greater than 1.

[0336] According to an embodiment of the present disclosure, the plurality of first configuration controls further include an image style configuration control.

[0337] According to an embodiment of the present disclosure, the second identification module 1220 includes a fourth identification submodule, a fifth identification submodule, and a sixth identification submodule.

[0338] The fourth identification submodule is configured to obtain image extension keywords based on the image style information returned by the image style configuration control.

[0339] The fifth recognition submodule is used to process the first original image using the visual language pre-training model to obtain at least one object text.

[0340] The fifth recognition submodule is configured to obtain an image description text based on at least one object text and image extension keywords.

[0341] According to an embodiment of the present disclosure, the plurality of first configuration controls further include a creativity configuration control.

[0342] According to an embodiment of the present disclosure, the second image generating module 1250 includes a fifth image generating submodule, a sixth image generating submodule, a seventh image generating submodule, an eighth image generating submodule, and a ninth image generating submodule.

[0343] The fifth image generation submodule is used to use a multimodal encoder to encode the second original image and the image description text respectively to obtain initial image encoding features and initial text encoding features.

[0344] The sixth image generation submodule is configured to modify the initial image coding features and the initial text coding features based on the creativity information returned by the creativity configuration control to obtain image coding features and text coding features.

[0345] The seventh image generation submodule is configured to initialize the first mask image using Gaussian noise to obtain a noise image.

[0346] The eighth image generation submodule is used to use the image coding features and text coding features as semantic benchmarks, use the image information creator to process the noisy image, and obtain the target output features.

[0347] The ninth image generation submodule is used to decode the target output features using the model decoder to obtain a first target mask image.

[0348] According to an embodiment of the present disclosure, the image processing apparatus 1200 further includes a second determining module and a first sending module.

[0349] The second determining module is configured to determine resolution information of the first original image in response to an image input request related to the first original image.

[0350] The first sending module is configured to send resolution information to the resolution display control, wherein the resolution display control is configured to display the original resolution of the first original image on the display interface based on the resolution information.

[0351] According to an embodiment of the present disclosure, the image processing apparatus 1200 further includes a second sending module.

[0352] The second sending module is configured to send the target extended image to the second image output control, wherein the second image output control is configured to display the target extended image on the display interface.

[0353] It should be noted that the image processing device part in another embodiment of the present disclosure corresponds to the image processing method part in another embodiment of the present disclosure. The description of the image processing device part specifically refers to the image processing method part and will not be repeated here.

[0354] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0355] For example, any multiple of the first recognition module 1110, the first generation module 1120, the first image generation module 1130, and the first stitching module 1140, or the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250, and the second stitching module 1260 can be combined into one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first recognition module 1110, the first generation module 1120, the first image generation module 1130 and the first stitching module 1140, or the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250 and the second stitching module 1260 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them. Alternatively, the first recognition module 1110, the first generation module 1120, the first image generation module 1130 and the first stitching module 1140, or at least one of the first acquisition module 1210, the second recognition module 1220, the second acquisition module 1230, the second generation module 1240, the second image generation module 1250 and the second stitching module 1260 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0356] According to an embodiment of the present disclosure, the image processing method or image processing device described above can also be used in various image generation application scenarios. For example, the image processing method can be applied in dynamic image generation scenarios, including but not limited to using the image processing method to generate template images of dynamic images, and using the image processing method to fine-tune images during the generation of dynamic image frames of different frames, etc., which are not limited here.

[0357] FIG13 schematically shows a flow chart of a method for generating a dynamic image according to an embodiment of the present disclosure.

[0358] As shown in FIG. 13 , the method 1300 includes operations S1310 to S1330 .

[0359] In operation S1310 , in response to the second task starting control being triggered, a prompt word text is obtained based on a return result of a prompt word input control included in the display interface.

[0360] In operation S1320 , the prompt word text is used as a semantic reference, and image generation is performed based on the initial template image to obtain a first template image.

[0361] In operation S1330 , a target dynamic image is generated based on the first template image according to a return result of the dynamic image configuration control included in the display interface.

[0362] According to an embodiment of the present disclosure, the second task start control may be a control for indicating the start of the dynamic image generation task, and the second task start control may be represented as a key or a button on the display interface. The user may touch and click the corresponding key or button, or the user may control the mouse to click the corresponding key or button to trigger the second task start control. When the second task start control is triggered, the electronic device may generate a template image generation request so that the client application process within the electronic device generates the first template image in response to the template image generation request.

[0363] According to an embodiment of the present disclosure, the prompt word input control can be an input box type control. The user can enter one or more prompt words or a piece of text in the input box of the prompt word input control. The prompt word input control can return the one or more prompt words or a piece of text as information. The electronic device can obtain the one or more prompt words or a piece of text by accessing the prompt word input control, and process the one or more prompt words or a piece of text to obtain the prompt word text. For example, when generating an image for product promotion, the user can enter prompt words or text about the product type, product location, background environment description, etc. in the input box of the prompt word input control, such as a round bottle on a stone, cloudy, grassland, trees, flowers, and a mountain peak faintly appearing in the distance.

[0364] According to an embodiment of the present disclosure, the second task starting control and the prompt word input control may be located on the same page of the display interface.

[0365] According to an embodiment of the present disclosure, the initial template image may be a blank image with a set resolution and aspect ratio.

[0366] According to an embodiment of the present disclosure, the method of generating a first template image based on a prompt word text can be implemented using the image processing method of an embodiment of the present disclosure, which will not be described in detail here.

[0367] According to embodiments of the present disclosure, the first template image can be fine-tuned to generate multiple dynamic image frames. Adjacent dynamic image frames generated through fine-tuning can have small differences between them. Playing these multiple dynamic image frames in sequence can achieve an animation effect. Therefore, by combining these multiple dynamic image frames, the target dynamic image can be generated.

[0368] The method shown in FIG. 13 will be further described below in conjunction with specific embodiments with reference to FIG. 14 to FIG. 16 , FIG. 17A to FIG. 17B , FIG. 18A to FIG. 18B , FIG. 19 and FIG. 20 .

[0369] FIG14 schematically shows a schematic diagram of a fourth display interface according to an embodiment of the present disclosure.

[0370] As shown in Figure 14, the fourth display interface can be represented as the portion of the display interface before starting the dynamic image generation task. This fourth display interface can include a prompt word input control, a second task start control, and an image display control. The prompt word input control can be represented as an input box in the fourth display interface, in which the user can enter image description text to complete the prompt word design. The second task start control can be represented as two buttons on the fourth display interface, including a "Start Generation" button for starting the task and a "Reset Settings" button for initializing the task. When performing the dynamic image generation task, if this is the first task after startup, the user can directly enter the image description text in the input box and then click the "Start Generation" button to begin generating the first template image and display the generated first template image in the image display control. If this is the second or more tasks, the user can first click the "Reset Settings" button to initialize the text content in the input box and the image content in the image display control, respectively, before entering the image description text and generating the first template image.

[0371] According to an embodiment of the present disclosure, the image description text entered by the user in the input box can be used as the return result of the prompt word input control. The electronic device can obtain the return result of the prompt word input control and split the image description text in the return result to obtain multiple prompt words. The multiple prompt words can each have their own type. Accordingly, the template text can include multiple blank fields, each of which can have its own type. The multiple prompt words can be written into the template text according to the type of each prompt word to obtain the prompt word text.

[0372] According to an embodiment of the present disclosure, the template text may be pre-configured by a developer. Alternatively, the electronic device may pre-store multiple template texts, and the electronic device may select an appropriate template text from the multiple template texts based on the semantic information of each of the multiple prompt words, or select a template text based on the return results of other controls.

[0373] According to an embodiment of the present disclosure, the fourth display interface may optionally further include an image style configuration control. The image style configuration control may be a selection item type configuration control, which may include multiple selection items representing image styles, such as a selection item representing an image style of "landscape," a selection item representing an image style of "starry sky," and a selection item representing an image style of "beach." The user may determine the image style information by selecting one of the selection items. This image style information may be used to determine an appropriate template text from a variety of template texts.

[0374] According to an embodiment of the present disclosure, the fourth display interface may optionally further include an aspect ratio configuration control. The aspect ratio configuration control may be a configuration control of a selection item type, and the aspect ratio configuration control may include multiple selection items representing the aspect ratio of the generated image, such as a selection item indicating that the aspect ratio of the generated image is "1:1", a selection item indicating that the aspect ratio of the generated image is "4:3", a selection item indicating that the aspect ratio of the generated image is "16:9", a selection item indicating that the aspect ratio of the generated image is "3:4", a selection item indicating that the aspect ratio of the generated image is "9:16", etc. The user may determine the aspect ratio of the generated first template image by selecting the above selection items.

[0375] According to an embodiment of the present disclosure, the fourth display interface may further include a first window for displaying the generated first template image.

[0376] According to embodiments of the present disclosure, the result returned by the prompt word input control can optionally be used directly as the prompt word text. In this case, the user needs to enter the complete prompt word text in the input box of the prompt word input control. For example, the user can enter "cloudy, grassland, trees, manifestation, mountains looming in the distance, a beautiful small jar on a stone." After clicking the "Start Generation" button, the first template image generated based on the user input text can be displayed in the first window.

[0377] According to an embodiment of the present disclosure, the fourth display interface may also include at least one interface jump control, which can be used to jump from the current fourth display interface to other display interfaces to complete subsequent operations on the first template image on other display interfaces.

[0378] According to an embodiment of the present disclosure, the at least one interface jump control may include an interface jump control "Dynamic Image Generation" for jumping to a fifth display interface. The fifth display interface may be the display interface when a user performs a dynamic image generation task. After the user clicks the interface jump control, the electronic device may send the generated first template image to a control within the fifth display interface, so that the control within the fifth display interface can be used to generate the image.

[0379] FIG15 schematically shows a schematic diagram of a fifth display interface according to an embodiment of the present disclosure.

[0380] As shown in FIG15 , similar to the fourth display interface, the fifth display interface may also include a task start control, which may include a “start generating” button for indicating starting a task and a “restore settings” button for indicating initializing a task.

[0381] According to an embodiment of the present disclosure, the fifth display interface may further include a dynamic image configuration control, which may be a selection item type configuration control. The dynamic image configuration control may include multiple selection items indicating the animation direction of the dynamic image to be generated, such as a selection item indicating that the animation direction of the generated dynamic image is "left-right," a selection item indicating that the animation direction of the generated dynamic image is "up-down," and a selection item indicating that the animation direction of the generated dynamic image is "circular," i.e., a viewing direction. The user may determine the animation effect of the target dynamic image to be generated by selecting the above selection items.

[0382] According to an embodiment of the present disclosure, the fifth display interface may further include a second window for displaying the image on which the dynamic image is generated, that is, the first template image, and a third window for playing the generated target dynamic image. After the user clicks on the interface jump control for jumping to the fifth display interface on the fourth display interface, the first template image generated in the fourth display interface can be transmitted to the fifth display interface and displayed in the second window. The user can select the desired animation direction from the multiple options in the dynamic image configuration control of the fifth display interface, and the information of the selected animation direction can be returned to the electronic device as the return result of the dynamic image configuration control. After the user clicks the "Start Generation" button, the electronic device can start generating the target dynamic image based on the first template image.

[0383] According to an embodiment of the present disclosure, the electronic device may generate a target dynamic image based on the first template image, including the following operations:

[0384] Performing depth prediction on the first template image to obtain a depth feature map of the first template image; splitting the first template image into multiple first connected domain images based on the depth feature map; generating multiple dynamic image frames based on the multiple first connected domain images according to the return result of the dynamic image configuration control; and combining the multiple dynamic image frames to obtain a target dynamic image.

[0385] According to an embodiment of the present disclosure, depth prediction of the first template image can be implemented using various depth prediction algorithms or models, which are not limited herein. The predicted depth feature map can be used to represent the depth information of each pixel in the first template image.

[0386] According to embodiments of the present disclosure, based on the depth information of each pixel in the image, pixels with similar depths expressed by the depth information and adjacent locations can be grouped into the same connected domain. That is, based on the depth information expressed in the depth feature map, multiple pixels contained in the first template image can be classified into multiple connected domains, and the pixels contained in each connected domain can be extracted to form a connected domain image.

[0387] According to an embodiment of the present disclosure, the return result of the dynamic image configuration control can represent the animation direction information of the target dynamic image to be generated. Based on the return result of the dynamic image configuration control, the processing strategy for each of the multiple first connected domain images associated with each dynamic image frame can be determined. The processing strategy for each of the multiple first connected domain images associated with each dynamic image frame can be different.

[0388] According to an embodiment of the present disclosure, the plurality of first connected domain images related to each dynamic image frame may be processed separately using respective processing strategies thereof to obtain each dynamic image frame, thereby generating multiple dynamic image frames.

[0389] According to an embodiment of the present disclosure, specifically, based on the processing strategy of each first connected domain image related to each frame of dynamic image, each first connected domain image can be translated, filled and / or rotated to obtain a fifth mask image corresponding to each first connected domain image; each first connected domain image is used as a semantic reference, and image generation is performed based on the fifth mask image corresponding to each first connected domain image to obtain multiple second connected domain images related to each frame of dynamic image; and the multiple second connected domain images related to each frame of dynamic image are spliced ​​to obtain each frame of dynamic image.

[0390] For example, the multiple first connected domain images may include a first connected domain image representing the foreground and a first connected domain image representing the background. When the animation direction information indicates "left," the first connected domain image representing the foreground may not be processed when generating each dynamic image frame. For each first connected domain image representing the background, a first connected domain image representing a shallower depth may be shifted rightward by a smaller number of pixels, while a first connected domain image representing a deeper depth may be shifted rightward by a larger number of pixels. The specific method for processing the multiple first connected domain images to obtain the multiple second connected domain images is not limited herein.

[0391] According to an embodiment of the present disclosure, optionally, after obtaining a plurality of second connected domain images, the plurality of second connected domains can be used as semantic references to generate each dynamic image frame based on the first template image, so as to repair overlapping areas or blank areas generated by splicing the plurality of second connected domain images.

[0392] According to an embodiment of the present disclosure, as an optional implementation, various processing operations may be performed on the first template image to obtain a new template image, and the new template image may be used to generate the target dynamic image. These various processing operations may include, but are not limited to, element replacement, image expansion, partial redrawing, etc. of the first template image, and are not limited here.

[0393] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as element replacement as an example, the user can use an existing reference image to replace the object element in the generated first template image to generate a new template image.

[0394] FIG16 schematically shows a schematic diagram of a fourth display interface according to another embodiment of the present disclosure.

[0395] As shown in Figure 16, the fourth display interface may further include an image input control, which may provide an image input interface, and the image input interface may be represented as a window on the fourth display interface. The user may click on the window to jump to the file selection interface to select and upload a reference image, or the user may drag the reference image to the window to upload the reference image. The electronic device may obtain a reference image based on the return result of the second image input control included in the display interface; and generate a second template image based on the first template image and the reference image. Thereafter, the target dynamic image may be generated based on the second template image according to the return result of the dynamic image configuration control included in the display interface.

[0396] According to an embodiment of the present disclosure, generating a second template image based on the first template image and the reference image may be to use the reference image to replace part of the image area in the first template image to obtain the second template image, or may be to use a specific object contained in the reference image to replace a specific object in the first template image.

[0397] According to an embodiment of the present disclosure, specifically, semantic segmentation can be performed on the first template image to obtain the first object included in the first template image and the position of the first object; and based on the position of the first object, the first object included in the first template image can be replaced with the second object included in the reference image to obtain the second template image.

[0398] According to an embodiment of the present disclosure, correspondingly, the second object contained in the reference image can also be obtained by performing semantic segmentation on the reference image.

[0399] FIG17A schematically shows a schematic diagram of a first template image according to an embodiment of the present disclosure.

[0400] FIG17B schematically shows a schematic diagram of a second template image according to an embodiment of the present disclosure.

[0401] As shown in Figures 17A and 17B , the size of the second object can be adjusted based on the size of the first object by scaling the second object so that its width and height are close to those of the first object and both are larger than those of the first object. Based on the position of the first object, the second object can be attached to the position of the first object in the first template image by aligning its bottom edges so that the second object completely covers the first object, thereby obtaining a second template image.

[0402] According to an embodiment of the present disclosure, the process of attaching the second object to the first object may cause jagged edges on the second object, resulting in a difference between the second object and the background. Optionally, after obtaining the second template image, the second template image may be partially redrawn to obtain a third template image, and then the target dynamic image may be generated based on the third template image according to the return result of the dynamic image configuration control included in the display interface.

[0403] According to an embodiment of the present disclosure, a third mask image may be generated based on the position of the second object in the second template image.

[0404] According to an embodiment of the present disclosure, optionally, when partially redrawing the second template image, an area to be redrawn may be determined first. The area to be redrawn may be the area where the second object is located in the second template image. Masking may be performed on the area to be redrawn to obtain a third mask image.

[0405] According to an embodiment of the present disclosure, the second template image can be binarized based on the position of the second object in the second template image to obtain an initial binarized image; the binarized image can be dilated and eroded to obtain a target binarized image; and the target binarized image can be superimposed on the second template image to obtain a third mask image.

[0406] According to an embodiment of the present disclosure, during the binarization process, the pixels in the area where the second object is located in the second template image can be assigned a value of 1, and the pixels in other areas of the second template image can be assigned a value of 0 to obtain an initial binarized image.

[0407] FIG18A schematically shows a schematic diagram of a target binarized image according to an embodiment of the present disclosure.

[0408] As shown in FIG18A , after obtaining the initial binarized image, the pixel points belonging to the area where the second object is located can be dilated respectively to expand the area where the second object is located outward, and the pixel points in the expanded area can all be assigned a value of 1. The area where the second object is located after the expansion can be eroded to assign a value of 0 to the pixel points in the partial area that is not related to the edge area of ​​the second object to obtain the target binarized image.

[0409] According to an embodiment of the present disclosure, before superimposing the target binary image onto the second template image, the target binary image may be subjected to inversion processing, i.e., multiple pixel points associated with multiple pixel points at the edge of the image belonging to the second object are all assigned a value of 0.

[0410] FIG18B schematically shows a schematic diagram of a third mask image according to an embodiment of the present disclosure.

[0411] As shown in FIG18B , the target binary image after color inversion processing can be superimposed on the second template image. After color inversion processing, only the pixel values ​​in the edge area of ​​the second object are 0, and the pixel values ​​in other areas are all 1. That is, the color of the pixel points in the edge area of ​​the second object is represented as black, and the color of the pixel points in other areas is represented as white. When the target binary image after color inversion processing is superimposed on the second template image, the color of the pixel points in the other areas obtained after superposition can be represented as the color of the pixel points of the original second template image, and the color of the pixel points in the edge area of ​​the second object can be represented as the color of the pixel points of the original target binary image, thereby obtaining the third mask image.

[0412] According to an embodiment of the present disclosure, the second template image may be used as a semantic reference, and image generation may be performed based on the third mask image to obtain a third template image.

[0413] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as a partial redrawing as an example, the fourth display interface shown in Figure 14 or Figure 16 may also include an interface jump control "partial redraw" for jumping to the sixth display interface. The sixth display interface can be the display interface when the user performs a partial redrawing task of the image. After the user clicks the interface jump control, the electronic device can send the generated first template image to the control in the sixth display interface so as to redraw the image using the control in the sixth display interface. Optionally, the reconstructed third template image can also be subjected to a partial redrawing operation to obtain a new template image, and then the new template image is used to generate a new target dynamic image, which will not be described in detail here.

[0414] FIG19 schematically shows a schematic diagram of a sixth display interface according to an embodiment of the present disclosure.

[0415] As shown in FIG19 , similar to the fourth display interface, the sixth display interface may also include a task start control, and the task start control may include a “start generating” button for indicating starting a task.

[0416] According to an embodiment of the present disclosure, the sixth display interface may further include a third image output control, which may include a fourth window for displaying the image before redrawing, namely, the first template image. The electronic device may send the first template image to the third image output control, and the third image output control may display the first template image in a fourth window on the display interface.

[0417] According to an embodiment of the present disclosure, the fourth window can optionally be configured as a human-computer interaction window, that is, the user can perform a smear operation on the first template image in the fourth window through touch screen control, mouse control, etc. The user's smear operation can be returned to the electronic device as a return result of the third image output control. In response to the smear operation on the first template image, the electronic device can determine a target redrawing area and partially redraw the target redrawing area to obtain the fourth template image.

[0418] According to an embodiment of the present disclosure, specifically, in response to a smearing operation on a first template image, a fourth mask image can be generated based on the smearing area indicated by the smearing operation, that is, the target redrawing area; and with the first template image as a semantic reference, image generation can be performed based on the fourth mask image to obtain a fourth template image.

[0419] According to an embodiment of the present disclosure, the sixth display interface may further include a fifth window, and the fifth window may be used to display the redrawn image, that is, the fourth template image.

[0420] According to an embodiment of the present disclosure, the sixth display interface may further include a mode selection configuration control. This mode selection configuration control may be a selection item type configuration control. The user can select from multiple options within this mode selection configuration control to determine how the first template image is to be redrawn. For example, the mode selection configuration control may include an option for "Erase" and an option for "Redraw." When the user selects the option for "Erase," the electronic device may perform a redraw operation to erase the foreground elements within the target redraw area selected by the user. When the user selects the option for "Redraw," the electronic device may perform a redraw operation to replace the foreground elements within the target redraw area selected by the user with the foreground elements specified by the user. Optionally, the sixth display interface may further include a redraw description information input control. This redraw description information input control may be an input box type configuration control, i.e., it may include an input box. The user may enter redraw description text within this input box, for example, "a little bird." This redraw description text may be returned to the electronic device as a result of the redraw description information input control. The electronic device can obtain the redrawing description text based on the return result of the redrawing description information input control, and after generating a fourth mask image based on the smearing area indicated by the smearing operation, use the first template image and the redrawing description text as semantic references to generate an image based on the fourth mask image to obtain a fourth template image.

[0421] According to an embodiment of the present disclosure, the sixth display interface may also include at least one interface jump control, which can be used to jump from the current sixth display interface to other display interfaces, such as the fourth display interface and the fifth display interface mentioned above, so as to complete subsequent operations on the fourth template image in other display interfaces.

[0422] According to an embodiment of the present disclosure, optionally, the at least one interface jump control may include an interface jump control "dynamic image generation" for jumping to a fifth display interface, and the fifth display interface may be a display interface when the user performs a dynamic image generation task. After the user clicks on the interface jump control, the electronic device may send the generated fourth template image to the control within the fifth display interface, so as to generate a dynamic image using the control within the fifth display interface. When generating a dynamic image, the electronic device may generate a target dynamic image based on the fourth template image according to the return result of the dynamic image configuration control contained in the display interface, which will not be elaborated here.

[0423] According to an embodiment of the present disclosure, optionally, taking the processing operation performed on the first template image as image expansion as an example, the fourth display interface shown in Figure 14 or Figure 16 may also include an interface jump control "image expansion" for jumping to the seventh display interface. The seventh display interface can be the display interface when the user performs an image expansion task. After the user clicks the interface jump control, the electronic device can send the generated first template image to the control in the seventh display interface so as to use the control in the seventh display interface to expand the image. Optionally, the image expansion operation can also be performed on the reconstructed third template image, fourth template image, etc. to obtain a new template image, and then the new template image is used to generate a new target dynamic image, which will not be repeated here.

[0424] FIG20 schematically shows a schematic diagram of a seventh display interface according to an embodiment of the present disclosure.

[0425] As shown in FIG20 , similar to the fourth display interface, the seventh display interface may also include a task start control, which may include a “start generating” button for indicating starting a task and a “restore settings” button for indicating initializing a task.

[0426] According to an embodiment of the present disclosure, the seventh display interface may include multiple second configuration controls, each of which may include at least a horizontal resolution configuration control and a vertical resolution configuration control. Both the horizontal resolution configuration control and the vertical resolution configuration control may be input box-type configuration controls. A user may enter resolution information in the input boxes of the horizontal resolution configuration control and the vertical resolution configuration control, respectively. This resolution information may be provided to the electronic device as a return result from each of the multiple second configuration controls. The electronic device may perform image expansion on the first template image based on this resolution information to obtain a fifth template image. For example, a user may enter 1920 in the input box of the horizontal resolution configuration control and 1080 in the input box of the vertical resolution configuration control. The electronic device may determine that the resolution information expressed by the user is to adjust the image resolution to 1920×1080. Based on the resolution information indicating 1920×1080, the electronic device may perform image expansion on the first template image to obtain a fifth template image with a resolution of 1920×1080. Specifically, based on the resolution information, a sixth mask image related to the first template image can be generated, and then the first template image is used as a semantic reference, and image generation is performed based on the sixth mask image to obtain a third target mask image, and then the third target mask image is spliced ​​with the first template image to obtain a fifth template image.

[0427] According to an embodiment of the present disclosure, the seventh display interface may optionally further include a sixth window for displaying the pre-expansion image, i.e., the first template image, and a seventh window for displaying the post-expansion image, i.e., the fifth template image. After the image expansion of the first template image is completed, the resulting fifth template image may be displayed in the seventh window to present the fifth template image to the user.

[0428] According to an embodiment of the present disclosure, optionally, the multiple second configuration controls may also include an image style configuration control, which may be a configuration control of a selection item type, and the image style configuration control may include multiple selection items representing image styles, for example, a selection item representing an image style of "landscape", a selection item representing an image style of "starry sky", a selection item representing an image style of "poster", and the like. The user may determine the image style information by selecting the selection items. The image style information may be used as semantic information to guide the image expansion process, that is, in the image expansion process, after determining the sixth mask image related to the first template image, the first template image and the image style information may be used as a semantic reference, and image generation may be performed based on the sixth mask image to obtain a third target mask image, and then the third target mask image may be spliced ​​with the first template image to obtain a fifth template image.

[0429] According to an embodiment of the present disclosure, the seventh display interface may also include at least one interface jump control, which can be used to jump from the current seventh display interface to other display interfaces, such as the fourth display interface, fifth display interface or sixth display interface mentioned above, so as to complete subsequent operations on the fifth template image in other display interfaces.

[0430] According to an embodiment of the present disclosure, optionally, the at least one interface jump control may include an interface jump control "dynamic image generation" for jumping to a fifth display interface, and the fifth display interface may be the display interface when performing a dynamic image generation task. After the user clicks on the interface jump control, the electronic device may send the generated fifth template image to the control within the fifth display interface so that the dynamic image is generated using the control within the fifth display interface. When generating a dynamic image, the electronic device may generate a target dynamic image based on the fifth template image according to the return result of the dynamic image configuration control contained in the display interface, which will not be described in detail here. Optionally, the at least one interface jump control may include an interface jump control "partial redraw" for jumping to a sixth display interface, and the sixth display interface may be the display interface when performing a partial redraw task. After the user clicks on the interface jump control, the electronic device may send the generated fifth template image to the control within the sixth display interface so that the control within the sixth display interface is used to perform a partial redraw of the template image, which will not be described in detail here.

[0431] According to the embodiments of the present disclosure, by using the above method for generating a target dynamic image based on a template image, dynamic images of various styles can be generated by fine-tuning the template image, which can effectively reduce the generation cost of dynamic images and improve the flexibility of dynamic image generation.

[0432] FIG21 schematically shows a block diagram of a dynamic image generating apparatus according to an embodiment of the present disclosure.

[0433] As shown in FIG. 21 , the dynamic image generating apparatus 2100 may include a third acquisition module 2110 , a third image generating module 2120 and a dynamic image generating module 2130 .

[0434] The third acquisition module 2110 is configured to obtain the prompt word text based on the return result of the prompt word input control included in the display interface in response to the second task starting control being triggered.

[0435] The third image generation module 2120 is configured to generate an image based on the initial template image using the prompt word text as a semantic reference to obtain a first template image.

[0436] The dynamic image generation module 2130 is configured to generate a target dynamic image based on the first template image according to the return result of the dynamic image configuration control included in the display interface.

[0437] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a first dynamic image generation submodule, a second dynamic image generation submodule, a third dynamic image generation submodule, and a fourth dynamic image generation submodule.

[0438] The first dynamic image generation submodule is used to perform depth prediction on the first template image to obtain a depth feature map of the first template image.

[0439] The second dynamic image generation submodule is used to split the first template image into multiple first connected domain images based on the depth feature map.

[0440] The third dynamic image generation submodule is configured to generate multiple dynamic image frames based on the multiple first connected domain images according to the return result of the dynamic image configuration control.

[0441] The fourth dynamic image generation submodule is used to combine multiple dynamic image frames to obtain a target dynamic image.

[0442] According to an embodiment of the present disclosure, the third dynamic image generation submodule includes a first dynamic image generation unit and a second dynamic image generation unit.

[0443] The first dynamic image generating unit is configured to determine respective processing strategies of a plurality of first connected domain images associated with each dynamic image frame according to a return result of the dynamic image configuration control.

[0444] The second dynamic image generating unit is used to process the multiple first connected domain images respectively using respective processing strategies of the multiple first connected domain images related to each dynamic image frame to obtain each dynamic image frame, so as to generate multiple dynamic image frames.

[0445] According to an embodiment of the present disclosure, the second dynamic image generation unit includes a first dynamic image generation sub-unit, a second dynamic image generation sub-unit, and a third dynamic image generation sub-unit.

[0446] The first dynamic image generation subunit is configured to perform translation, filling and / or rotation processing on each first connected domain image based on a processing strategy for each first connected domain image associated with each dynamic image frame, to obtain a fifth mask image corresponding to each first connected domain image.

[0447] The second dynamic image generation subunit is used to use each first connected domain image as a semantic benchmark and perform image generation based on a fifth mask image corresponding to each first connected domain image to obtain multiple second connected domain images related to each dynamic image frame.

[0448] The third dynamic image generation subunit is used to splice multiple second connected domain images related to each dynamic image frame to obtain each dynamic image frame.

[0449] According to an embodiment of the present disclosure, the dynamic image generating apparatus 2000 may further include a fourth acquisition module and a fourth image generating module.

[0450] The fourth acquisition module is configured to obtain a reference image based on a return result of the second image input control included in the display interface.

[0451] The fourth image generating module is configured to generate a second template image based on the first template image and the reference image.

[0452] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a fourth dynamic image generation submodule.

[0453] The fourth dynamic image generation submodule is configured to generate a target dynamic image based on the second template image according to a return result of the dynamic image configuration control included in the display interface.

[0454] According to an embodiment of the present disclosure, the fourth image generating module includes a tenth image generating submodule and an eleventh image generating submodule.

[0455] The tenth image generation submodule is configured to perform semantic segmentation on the first template image to obtain the first object included in the first template image and the position of the first object.

[0456] The eleventh image generation submodule is configured to replace the first object included in the first template image with the second object included in the reference image based on the position of the first object, so as to obtain a second template image.

[0457] According to an embodiment of the present disclosure, the dynamic image generating apparatus 2100 may further include a first redrawing module.

[0458] The first redrawing module is used to partially redraw the second template image to obtain a third template image.

[0459] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a fifth dynamic image generation submodule.

[0460] The fifth dynamic image generation submodule is configured to generate a target dynamic image based on the third template image according to a return result of the dynamic image configuration control included in the display interface.

[0461] According to an embodiment of the present disclosure, the first redrawing module includes a first redrawing sub-module and a second redrawing sub-module.

[0462] The first redrawing submodule is configured to generate a third mask image based on a position of the second object in the second template image.

[0463] The second redrawing submodule is configured to use the second template image as a semantic reference and generate an image based on the third mask image to obtain a third template image.

[0464] According to an embodiment of the present disclosure, the first redrawing submodule includes a first redrawing unit, a second redrawing unit, and a third redrawing unit.

[0465] The first redrawing unit is configured to perform a binarization process on the second template image based on the position of the second object in the second template image to obtain an initial binarized image.

[0466] The second redrawing unit is used to perform expansion and corrosion processing on the binary image to obtain a target binary image.

[0467] The third redrawing unit is used to superimpose the target binary image on the second template image to obtain a third mask image.

[0468] According to an embodiment of the present disclosure, the dynamic image generating apparatus 2100 may further include a third sending module and a second redrawing module.

[0469] The third sending module is configured to send the first template image to the third image output control, wherein the third image output control is configured to display the first template image on the display interface.

[0470] The second redrawing module is configured to partially redraw the first template image in response to the smearing operation on the first template image to obtain a fourth template image.

[0471] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a sixth dynamic image generation submodule.

[0472] The sixth dynamic image generation submodule is configured to generate a target dynamic image based on the fourth template image according to a return result of the dynamic image configuration control included in the display interface.

[0473] According to an embodiment of the present disclosure, the second redrawing module includes a third redrawing submodule and a fourth redrawing submodule.

[0474] The third redrawing submodule is configured to generate a fourth mask image in response to a smearing operation on the first template image and based on a smearing area indicated by the smearing operation.

[0475] The fourth redrawing submodule is configured to use the first template image as a semantic reference and perform image generation based on the fourth mask image to obtain a fourth template image.

[0476] According to an embodiment of the present disclosure, the display interface further includes a redraw description information input control.

[0477] According to an embodiment of the present disclosure, the fourth redrawing submodule includes a fourth redrawing unit and a fifth redrawing unit.

[0478] The fourth redrawing unit is used to obtain the redrawing description text based on the return result of the redrawing description information input control.

[0479] The fifth redrawing unit is configured to use the first template image and the redrawing description text as semantic references and perform image generation based on the fourth mask image to obtain a fourth template image.

[0480] According to an embodiment of the present disclosure, the dynamic image generating apparatus 2100 may further include a fifth image generating module.

[0481] The fifth image generating module is configured to perform image expansion on the first template image based on the return results of each of the plurality of second configuration controls included in the display interface to obtain a fifth template image.

[0482] According to an embodiment of the present disclosure, the dynamic image generation module 2130 includes a seventh dynamic image generation submodule.

[0483] The seventh dynamic image generation submodule is configured to generate a target dynamic image based on the fifth template image according to a return result of the dynamic image configuration control included in the display interface.

[0484] According to an embodiment of the present disclosure, the third acquisition module 2110 includes a third acquisition submodule and a fourth acquisition submodule.

[0485] The third acquisition submodule is used to obtain multiple prompt words based on the return result of the prompt word input control.

[0486] The fourth acquisition submodule is used to write multiple prompt words into the template text respectively to obtain the prompt word text.

[0487] It should be noted that the dynamic image generation device part in the embodiment of the present disclosure corresponds to the dynamic image generation method part in the embodiment of the present disclosure. The description of the dynamic image generation device part specifically refers to the dynamic image generation method part and will not be repeated here.

[0488] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0489] For example, any multiple of the third acquisition module 2110, the third image generation module 2120, and the dynamic image generation module 2130 can be combined into one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the third acquisition module 2110, the third image generation module 2120, and the dynamic image generation module 2130 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the third acquisition module 2110, the third image generation module 2120, and the dynamic image generation module 2130 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed. Figure 22 schematically shows a block diagram of an electronic device suitable for implementing the image processing method or the dynamic image generation method according to an embodiment of the present disclosure. The electronic device shown in Figure 22 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0490] As shown in Figure 22, the computer electronic device 2200 according to an embodiment of the present disclosure includes a processor 2201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2202 or the program loaded from the storage part 2208 into the random access memory (RAM) 2203. The processor 2201 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 2201 may also include onboard memory for caching purposes. The processor 2201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0491] The RAM 2203 stores various programs and data required for the operation of the electronic device 2200. The processor 2201, the ROM 2202, and the RAM 2203 are connected to each other via a bus 2204. The processor 2201 executes the programs in the ROM 2202 and / or the RAM 2203 to perform various operations of the method flow according to the embodiment of the present disclosure. It should be noted that the programs may also be stored in one or more memories other than the ROM 2202 and the RAM 2203. The processor 2201 may also execute the programs stored in the one or more memories to perform various operations of the method flow according to the embodiment of the present disclosure.

[0492] According to an embodiment of the present disclosure, electronic device 2200 may further include an input / output (I / O) interface 2205, which is also connected to bus 2204. Electronic device 2200 may further include one or more of the following components connected to I / O interface 2205: an input section 2206 including a keyboard, mouse, etc.; an output section 2207 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 2208 including a hard disk; and a communication section 2209 including a network interface card such as a LAN card or modem. Communication section 2209 performs communication processing via a network such as the Internet. A drive 22222210 is also connected to I / O interface 2205 as needed. Removable media 2211, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 22222210 as needed, so that computer programs read from the removable media can be installed into storage section 2208 as needed.

[0493] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 2209, and / or installed from the removable medium 2211. When the computer program is executed by the processor 2201, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0494] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0495] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0496] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 2202 and / or the RAM 2203 described above and / or one or more memories other than the ROM 2202 and the RAM 2203 .

[0497] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present disclosure.

[0498] When the computer program is executed by the processor 2201, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0499] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 2209, and / or installed from the removable medium 2211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0500] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0501] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0502] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method, comprising: In response to the image expansion request, performing visual language recognition on the first original image to obtain an image description text; Generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution included in the image extension request; Using the image description text and the second original image as semantic references, and performing image generation based on the first mask image to obtain a first target mask image; as well as The first target mask image and the second original image are stitched together to obtain a target extended image.

2. The method according to claim 1, wherein: The step of using the target aspect ratio and the first target resolution included in the image extension request to generate a second original image and a first mask image based on the first original image includes: Based on the target aspect ratio and the first target resolution, performing image expansion with at least one edge of the first original image as a starting point to obtain a first expanded image; and The second original image and the first mask image are determined from the first extended image, wherein the second original image is the first original image.

3. The method according to claim 1, wherein: The step of using the target aspect ratio and the first target resolution included in the image extension request to generate a second original image and a first mask image based on the first original image includes: Based on the original resolution and the preset resolution of the first original image, scaling the first original image to obtain a third original image; Based on the target aspect ratio, performing image expansion with at least one side of the third original image as a starting point to obtain a second expanded image; and When the first target resolution is equal to the preset resolution, the second original image and the first mask image are determined from the second extended image.

4. The method according to claim 3, further comprising: When the first target resolution is not equal to the preset resolution, scaling the first extended image based on the first target resolution and the preset resolution to obtain a third extended image; as well as The second original image and the first mask image are determined from the third extended image.

5. The method according to any one of claims 1 to 4, further comprising: Based on the first target resolution and the original resolution of the first original image, determining that the preset conditions are met In the case of the first original image and the image description text as semantic references, image expansion is performed N times to obtain the target expanded image, wherein N is a positive integer; The preset condition includes any one of the following: A ratio between the first target resolution and the original resolution of the first original image is less than a first threshold; A ratio between the first target resolution and the original resolution of the first original image is greater than a second threshold; and The image extension request includes an image extension number greater than 1.

6. The method according to claim 5, wherein: The step of performing N times of image expansion using the first original image and the image description text as semantic references to obtain the target expanded image includes: Determine an expansion ratio based on the first target resolution, the original resolution of the first original image, and N; In a case where the first target resolution and the image resolution of the extended image after the i-1th image expansion are not equal, determining the second target resolution based on the image resolution of the extended image after the i-1th image expansion and the expansion magnification, wherein i is a positive integer less than or equal to N, the extended image after the 0th image expansion is represented as the first original image, and the image resolution of the extended image after the 0th image expansion is represented as the original resolution; Generate a fourth original image and a second mask image based on the extended image after the (i-1)th image expansion by using the target aspect ratio and the second target resolution; Using the image description text and the fourth original image as semantic references, performing image generation based on the second mask image to obtain a second target mask image; and The fourth original image and the second target mask image are stitched together to obtain an extended image after the i-th image expansion.

7. The method according to claim 6, further comprising: In a case where the first target resolution is equal to the image resolution of the extended image after the i-th image expansion, the extended image after the i-th image expansion is determined as the target extended image.

8. The method according to claim 1, wherein: The step of taking the image description text and the second original image as semantic references and performing image generation based on the first mask image to obtain a first target mask image includes: Using a multimodal encoder to encode the second original image and the image description text respectively to obtain image encoding features and text encoding features; Initializing the first mask image using Gaussian noise to obtain a noise image; Using the image coding features and text coding features as semantic benchmarks, using the image information creator to process the noisy image to obtain target output features; and The target output features are decoded using a model decoder to obtain the first target mask image.

9. The method according to claim 8, wherein: The image information creator includes an encoding module, a decoding module and an intermediate module; The method of using the image coding features and the text coding features as semantic benchmarks and using the image information creator to process the noise image to obtain target output features includes: Using the encoding module to process the image encoding feature, the text encoding feature and the noise image, obtain a first noise image feature output by each of the M encoding function blocks included in the encoding module, wherein M is a positive integer; Processing the first noise image feature output by the Mth encoding function block using the intermediate module to obtain a second noise image feature; and The target output feature is obtained by processing the second noise image feature, the first noise image feature outputted by each of the M encoding function blocks, the image encoding feature and the text encoding feature using the decoding module.

10. The method according to claim 9, wherein: The M encoding function blocks are connected via cross attention blocks; The step of using the encoding module to process the image encoding feature, the text encoding feature and the noise image to obtain the first noise image feature output by each of the M encoding function blocks included in the encoding module includes: Processing the input feature of the j-th encoding function block by using the j-th encoding function block to obtain a first noise image feature output by the j-th encoding function block, wherein j is a positive integer less than or equal to M, and the input feature of the first encoding function block is represented as the noise image; and When j is less than M, the cross-attention block between the j-th encoding functional block and the j+1-th encoding functional block is used to process the first noise image feature, the image encoding feature, and the text encoding feature output by the j-th encoding functional block to obtain the input feature of the j+1-th encoding functional block.

11. The method according to claim 9, wherein: The decoding module includes M decoding function blocks, and the M decoding function blocks are connected by cross attention blocks; wherein the decoding module is used to process the second noise image feature, the first noise image feature output by each of the M encoding function blocks, the image encoding feature and the text encoding feature to obtain the target Standard output features include: Using the kth decoding function block to process the input feature of the kth decoding function block and the first noise image feature output by the kth encoding function block, to obtain a decoding feature output by the kth decoding function block, wherein k is a positive integer less than or equal to M, and the input feature of the first decoding function block is represented as the second noise image feature; In the case where k is less than M, using a cross attention block between the kth decoding function block and the k+1th decoding function block to process the decoding features output by the kth decoding function block, the image coding features, and the text coding features, to obtain input features of the k+1th decoding function block; and When k is equal to M, the target output feature is determined to be a decoding feature output by the kth decoding functional block.

12. The method according to any one of claims 9 to 11, wherein: The intermediate module, the encoding functional block and the decoding functional block are gated convolution blocks; The step of using the intermediate module to process the first noise image feature output by the Mth encoding function block to obtain the second noise image feature includes: Performing layer normalization processing on the first noise image feature output by the Mth encoding function block to obtain a first sub-feature; Processing the first sub-feature using a gated convolutional layer to obtain a second sub-feature; Performing layer normalization processing on the second sub-feature to obtain a third sub-feature; Processing the third sub-feature using a perceptron to obtain a fourth sub-feature; and The second noise image feature is obtained based on the first noise image feature output by the Mth encoding function block and the fourth sub-feature.

13. The method according to claim 12, wherein: The gated convolution layer includes P gated processing units, where P is a positive integer; The step of processing the first sub-feature using a gated convolutional layer to obtain a second sub-feature includes: Performing convolution processing on the first sub-feature to obtain a fifth sub-feature; Performing a deep convolution process on the fifth sub-feature to obtain a deep convolution feature; For the p-th gated processing unit, performing a dot multiplication process on the deep convolution feature and the gated output feature output by the p-1-th gated processing unit to obtain a sixth sub-feature of the p-th gated processing unit, wherein p is a positive integer less than or equal to P; and The sixth sub-feature of the p-th gate processing unit is convolved to obtain the p-th gate processing unit. The gated output feature of the 0th gated processing unit is represented as the fifth sub-feature, and the gated output feature of the Pth gated processing unit is represented as the second sub-feature.

14. The method according to claim 1, wherein: The step of performing visual language recognition on the first original image to obtain image description text includes: Processing the first original image using a visual language pre-trained model to obtain at least one object text; and The image description text is obtained based on the at least one object text.

15. The method according to claim 14, wherein: The image extension request also includes an image extension keyword; The method further comprises: The image description text is obtained based on the at least one object text and the image extension keyword.

16. An image processing method, comprising: In response to the first task start control being triggered, acquiring a first original image from a return result of a first image input control included in the display interface; Performing visual language recognition on the first original image to obtain image description text; Based on the returned results of each of the plurality of first configuration controls included in the display interface, a target aspect ratio and a first target resolution are obtained; Generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution; Using the image description text and the second original image as semantic references, and performing image generation based on the first mask image to obtain a first target mask image; as well as The first target mask image and the second original image are stitched together to obtain a target extended image.

17. The method according to claim 16, wherein: The plurality of first configuration controls include a horizontal resolution configuration control, a vertical resolution configuration control, a horizontal expansion ratio configuration control, and a vertical resolution configuration control; Wherein, obtaining the target aspect ratio and the first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface includes: Determining the first target resolution based on the horizontal resolution information returned by the horizontal resolution configuration control, the vertical resolution information returned by the vertical resolution configuration control, the horizontal expansion ratio information returned by the horizontal expansion ratio configuration control, and the vertical expansion ratio information returned by the vertical resolution configuration control; and The target aspect ratio is determined based on the first target resolution.

18. The method according to claim 16, wherein: The plurality of first configuration controls also include an expansion direction configuration control; The step of generating a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution includes: Determining at least one target edge from four edges of the first original image based on the extension direction information returned by the extension direction configuration control; Based on the target aspect ratio and the first target resolution, performing image expansion with the at least one target edge of the first original image as a starting point to obtain a first expanded image; and The second original image and the first mask image are determined from the first extended image, wherein the second original image is the first original image.

19. The method according to claim 18, wherein: The plurality of first configuration controls also include a candidate image quantity control; The step of performing image expansion based on the target aspect ratio and the first target resolution and taking the at least one target edge of the first original image as a starting point to obtain a first expanded image includes: Determine the number of candidate images based on the candidate image number information returned by the candidate image number control; Based on the target aspect ratio and the first target resolution, performing image expansion with the at least one target edge of the first original image as a starting point to obtain fifth expanded images of the number of candidate images; Sending a fifth expanded image of the number of candidate images to a first image output control, wherein the first image output control is configured to display the fifth expanded image of the number of candidate images on the display interface; and In response to a selection request for a fifth extended image of the number of candidate images, the first extended image is determined from the fifth extended images of the number of candidate images.

20. The method according to claim 16, wherein: The plurality of first configuration controls also include a generation speed configuration control and a single maximum expansion pixel control; The method further comprises: Determining the number of image expansion times based on the generation speed information returned by the generation speed configuration control, the single maximum expansion pixel information returned by the single maximum expansion pixel control, the original resolution of the first original image, and the first target resolution; and When the number of image expansion times is greater than 1, based on the number of image expansion times, multiple image expansions are performed with the first original image and the image description text as semantic references to obtain the target expanded image.

21. The method according to claim 16, wherein: The plurality of first configuration controls also include an image style configuration control; The performing visual language recognition on the first original image to obtain image description text includes: Obtaining image extension keywords based on the image style information returned by the image style configuration control; Processing the first original image using a visual language pre-trained model to obtain at least one object text; and The image description text is obtained based on the at least one object text and the image extension keyword.

22. The method according to claim 16, wherein: The plurality of first configuration controls further include a creativity configuration control; The step of taking the image description text and the second original image as semantic references and performing image generation based on the first mask image to obtain a first target mask image includes: Using a multimodal encoder to encode the second original image and the image description text respectively to obtain initial image encoding features and initial text encoding features; Based on the creativity information returned by the creativity configuration control, the initial image coding feature and the initial text coding feature are modified to obtain an image coding feature and a text coding feature; Initializing the first mask image using Gaussian noise to obtain a noise image; Using the image coding features and text coding features as semantic benchmarks, using the image information creator to process the noisy image to obtain target output features; and The target output features are decoded using a model decoder to obtain the first target mask image.

23. The method of claim 16, further comprising: In response to an image input request associated with the first original image, determining resolution information of the first original image; as well as The resolution information is sent to a resolution display control, wherein the resolution display control is configured to display an original resolution of the first original image on the display interface based on the resolution information.

24. The method of claim 16, further comprising: The target extended image is sent to a second image output control, wherein the second image output control is configured to display the target extended image on the display interface.

25. An image processing device, comprising: A first recognition module, configured to perform visual language recognition on the first original image in response to the image expansion request to obtain an image description text; A first generating module, configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution included in the image extension request; A first image generation module, configured to use the image description text and the second original image as semantic references and to generate an image based on the first mask image to obtain a first target mask image; as well as The first stitching module is used to stitch the first target mask image with the second original image to obtain a target extended image.

26. An image processing device, comprising: A first acquisition module, configured to acquire a first original image from a return result of a first image input control included in the display interface in response to the first task start control being triggered; A second recognition module is used to perform visual language recognition on the first original image to obtain an image description text; A second acquisition module, configured to obtain a target aspect ratio and a first target resolution based on the return results of each of the plurality of first configuration controls included in the display interface; a second generating module, configured to generate a second original image and a first mask image based on the first original image by using the target aspect ratio and the first target resolution; A second image generation module, configured to use the image description text and the second original image as semantic references and to generate an image based on the first mask image to obtain a first target mask image; as well as The second stitching module is used to stitch the first target mask image with the second original image to obtain a target extended image.

27. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 24.

28. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 24.

29. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 24 when the computer executable instructions are executed.

Images