Image processing method and device, equipment, storage medium and program product

Abstract

The invention provides an image processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the fusion of a foreground coding vector in an obtained to-be-generated image and a scene coding vector used for indicating a scene where the to-be-generated image is located, and generating a first target image. The interaction relationship between the foreground in the to-be-generated image and other objects in the scene in the vector space can be reflected through the fusion of the foreground coding vector and the scene coding vector, so that the fusion between the foreground in the generated image and other objects is more natural.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of e-commerce platforms, product display has become an important way for users to understand products. To expand product usage scenarios, merchants often replace the backgrounds of model or product photos taken in studios to show the visual effects of the same type of product in different scenarios.

[0003] Currently, background replacement for model and product images after capture primarily relies on image editing tools like Photoshop (PS). This method involves manually cutting out the foreground, pasting it onto a selected background, and performing detailed processing. However, this method not only requires operator skill, but also struggles to achieve a natural blend between the foreground and the new background, resulting in an abrupt appearance in the resulting image. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image processing method, apparatus, device, storage medium and program product.

[0005] According to a first aspect of the present disclosure, there is provided an image generation method, the method comprising:

[0006] Obtain a foreground coding vector of a first foreground mask image; obtain a scene coding vector for indicating a scene where an image to be generated is located; and fuse the foreground coding vector and the scene coding vector to generate a first target image.

[0007] In some embodiments, obtaining the foreground coding vector of the first foreground mask image includes: obtaining the first foreground mask image; and encoding the first foreground mask image to obtain the foreground coding vector of the first foreground mask image.

[0008] In some embodiments, obtaining a first foreground mask image includes: in response to receiving a first image containing a background input by a user, extracting a foreground mask of a foreground object in the first image through a pre-trained foreground mask acquisition model, wherein the foreground mask acquisition model is trained through multiple training sample images, and each training sample image is annotated with the foreground mask of the image and a border surrounding the location of the foreground mask; generating a first foreground mask image according to the position of the foreground mask in the first image.

[0009] In some embodiments, obtaining the first foreground mask image includes: in response to receiving a second image containing a background and a specified area of ​​interest input by a user, identifying the subject where the area of ​​interest is located in the second image; and segmenting the subject in the second image to obtain the first foreground mask image.

[0010] In some embodiments, the scene encoding vector comprises a first encoding vector;

[0011] The obtaining of a scene coding vector for indicating a scene in which the image to be generated is located includes: in response to receiving text information input by a user, encoding the text information using a text encoder to generate a first coding vector, wherein the text information is used to describe the scene of the image to be generated.

[0012] In some embodiments, the scene encoding vector includes a second encoding vector;

[0013] The obtaining of a scene coding vector for indicating a scene in which the image to be generated is located includes: in response to receiving a reference image input by a user, encoding the reference image using an image encoder to generate a second coding vector, wherein the reference image is used to represent the scene of the image to be generated.

[0014] In some embodiments, the scene encoding vector includes a third encoding vector;

[0015] The method of obtaining a scene coding vector indicating a scene in which an image to be generated is located includes: in response to receiving text information and a reference image input by a user, encoding the text information using a text encoder to generate a first coding vector, and encoding the reference image using an image encoder to generate a second coding vector; and fusing the first coding vector and the second coding vector to obtain a third coding vector.

[0016] In some embodiments, the method further comprises: acquiring a control vector corresponding to a control condition input by a user, the control condition comprising at least one of a posture graph, a scene layout reference graph, and a light source distribution graph;

[0017] The fusing of the foreground coding vector and the scene coding vector to generate a control vector corresponding to the first target image acquisition layout control condition includes:

[0018] The foreground coding vector, the scene coding vector and the control vector are fused to generate a first target image.

[0019] In some embodiments, obtaining a control vector corresponding to a control condition input by a user includes: obtaining the control condition input by the user; and encoding at least one obtained control condition to generate a control vector.

[0020] In some embodiments, when the foreground mask image contains a target object, the control condition includes a posture graph, wherein the posture graph is used to represent the posture of the target object in the foreground mask image through a skeleton;

[0021] The obtaining of the control condition input by the user includes: obtaining a first posture diagram drawn by the user according to a first preset rule; or obtaining a second posture diagram selected by the user from a reference posture diagram.

[0022] In some embodiments, when the scene coding vector is obtained through text information, the control condition includes a scene layout reference map, where the scene layout reference map is used to indicate the spatial position of each object in the scene;

[0023] The obtaining of the control condition input by the user includes: obtaining a first scene layout reference diagram drawn by the user according to a second preset rule; or obtaining a second scene layout reference diagram selected by the user from the scene layout example diagram.

[0024] In some embodiments, the control conditions further include a light source distribution diagram;

[0025] The obtaining of the control condition input by the user includes: obtaining a light source distribution map selected by the user from a reference light source distribution map.

[0026] In some embodiments, after generating the first target image, the method further includes: in response to receiving a modification instruction, displaying the control conditions for generating the first target image; modifying the control conditions according to the user's operation information; and in response to receiving a generation instruction, generating a corresponding target image according to the modified control conditions.

[0027] In some embodiments, after generating the first target image, the method further includes: obtaining a foreground coding vector of a second foreground mask image, the second foreground mask image having the same scene as the first foreground mask image; and generating a second target image based on the foreground coding vector of the second foreground mask image, the scene coding vector and the control vector.

[0028] According to a second aspect of the present disclosure, there is provided an image generating apparatus, the apparatus comprising:

[0029] a foreground processing module, configured to obtain a foreground coding vector of the first foreground mask image;

[0030] A scene processing module, used to obtain a scene coding vector indicating the scene of the image to be generated;

[0031] The image generation module is used to fuse the foreground coding vector and the scene coding vector to generate a first target image.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, so as to execute the method described in any one of the above embodiments.

[0033] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0034] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program and instructions, wherein when the computer program / instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0035] The technical solutions provided by the present disclosure may have the following beneficial effects:

[0036] The present disclosure generates a first target image by fusing a foreground coding vector obtained from an image to be generated with a scene coding vector indicating the scene in which the image to be generated resides. The fusion of the foreground coding vector and the scene coding vector can reflect the interactive relationship between the foreground of the image to be generated and other objects in the scene in vector space, thereby achieving a more natural fusion between the foreground and other objects in the generated image.

[0037] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, are used to explain the technical solutions of the present disclosure.

[0039] Figure 1 This is a flow chart of an image generation method according to an exemplary embodiment of the present disclosure.

[0040] Figure 2 is a schematic diagram of a scene processing module according to an exemplary embodiment of the present disclosure.

[0041] Figure 3 is a schematic diagram of a reference image according to an exemplary embodiment of the present disclosure.

[0042] Figure 4 is a schematic diagram of a posture diagram according to an exemplary embodiment of the present disclosure.

[0043] Figure 5 It is a schematic diagram of a scene layout reference diagram according to an exemplary embodiment of the present disclosure.

[0044] Figure 6 is a schematic diagram of a light source distribution diagram according to an exemplary embodiment of the present disclosure.

[0045] Figure 7 It is a framework diagram of an image generating device according to an exemplary embodiment of the present disclosure.

[0046] Figure 8 The figure is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0048] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0050] When shopping online, since users can't touch the products, the visual quality of product images becomes a crucial factor in selecting products. Merchants need to provide images that reflect the product's characteristics and appeal to users. Since the same product can be used in different scenarios, to enrich the product presentation, existing model and product images can be replaced with different scenes, creating product displays with diverse styles and themes.

[0051] In the related art, scene replacement can be performed manually using image editing tools, but this method has at least the following problems:

[0052] (1) Manually selecting foreground objects and performing accurate segmentation takes a lot of time.

[0053] (2) There are high skill requirements for operators.

[0054] (3) Manual scene replacement results in an unnatural fusion of the foreground area and the new background area in the generated image.

[0055] In view of this, the present disclosure provides an image generation method that can automatically generate a scene for the foreground that meets the scene coding vector requirements based on the obtained foreground coding vector and scene coding vector. First, generating images through the present disclosure can significantly reduce the investment of manpower and material resources, thereby improving production efficiency. Second, for operators using the present disclosure, there is no need to master tedious image editing skills. By providing a simple scene description or the required scene reference image, the scene replacement operation in the model image and the product image can be completed.

[0056] The following embodiments will introduce the method provided by the present disclosure in conjunction with the accompanying drawings.

[0057] Figure 1 This is a flow chart of an image generation method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method may include the following steps 101 to 104.

[0058] In step 101, a foreground encoding vector of a first foreground mask image is obtained.

[0059] The first foreground mask image is the foreground of the image to be generated. The foreground can be a model, such as a person or an animal; the foreground can also be a commodity, such as daily necessities or toiletries. In this embodiment, an existing encoder can be used to obtain the foreground encoding vector of the first foreground mask image.

[0060] In step 102, a scene coding vector indicating the scene in which the image to be generated is located is obtained.

[0061] In step 103, the foreground coding vector and the scene coding vector are fused to generate a first target image.

[0062] After fusing the foreground coding vector and the scene coding vector, the interactive relationship between the foreground and other objects in the scene can be reflected in space. Therefore, not only does the scene in the generated image meet the requirements of the scene coding vector, but the foreground in the image and other objects in the scene can also be naturally integrated.

[0063] In this embodiment, the foreground coding vector and the scene coding vector can be input into a pre-trained image generation model, and the image generation model can be used to generate the first target image. The image generation model can adopt diffusion models (DM), generative adversarial networks (GAN), variational autoencoders (VAE), etc., which are not limited in this disclosure.

[0064] The present disclosure generates a first target image by fusing a foreground coding vector in an image to be generated with a scene coding vector indicating the scene in which the image to be generated resides. The fusion of the foreground coding vector and the scene coding vector can reflect the interactive relationship between the foreground in the image to be generated and other objects in the scene in vector space, thereby making the fusion between the foreground and other objects in the generated image more natural.

[0065] In some embodiments, obtaining the foreground coding vector of the first foreground mask image may include: obtaining the first foreground mask image; and encoding the first foreground mask image to obtain the foreground coding vector of the first foreground mask image.

[0066] In one embodiment, a user-specified image to be processed may be obtained, and the image to be processed may be a model image or a product image, etc. The image to be processed may be an image selected locally by the user, an image downloaded from the cloud, or an image received from another device.

[0067] The image to be processed may be a foreground image containing a background for which generation is required, or a foreground image for which background replacement is required. Specifically, if the user-specified image to be processed is a four-channel image containing a foreground region mask, the user-specified image to be processed serves as the first foreground mask image. If the user-specified image to be processed already has a background for which background replacement is required, the first foreground mask image may be obtained from the user-specified image to be processed using the following embodiments.

[0068] When a first image containing a background is received from a user, a foreground mask of a foreground object in the first image can be extracted using a pre-trained foreground mask acquisition model. The foreground mask acquisition model is trained using multiple training sample images, each of which is annotated with the foreground mask of the image and a border surrounding the location of the foreground mask. A first foreground mask image is generated based on the position of the foreground mask in the first image.

[0069] When the user inputs a first image containing a background, a foreground mask acquisition model such as a segmentation or matting model may be used to acquire a foreground mask region.

[0070] In this embodiment, the foreground mask area can not only be obtained by model prediction, but also can be a foreground part that needs to be retained specified by the user.

[0071] That is, when a second image including a background and a designated region of interest is received from a user, a subject where the region of interest is located is identified in the second image; the subject is segmented in the second image to obtain a first foreground mask image.

[0072] To improve the accuracy of foreground extraction, the user can define the foreground area in the second image with a bounding box. The area within the bounding box is referred to as the user-specified region of interest. Upon receiving a second image containing a background and a specified region of interest from the user, the subject within the region of interest can be identified in the second image. An image segmentation algorithm, such as the GrabCut algorithm or a deep learning-based semantic segmentation model, is then used to segment the subject in the second image, generating a first foreground mask image.

[0073] As can be seen from the above embodiments, this embodiment provides users with multiple ways to obtain foreground mask images. For example, if the user chooses to provide an image with an existing background, the present invention can automatically obtain a foreground mask image from the existing background image. If the user specifies a foreground portion in an image with an existing background, the present invention can segment the foreground mask image from the image based on the user-specified foreground portion. This highly customizable approach allows the generated scene to better meet the user's individual needs and provides an accurate foreground mask image for subsequent scene generation operations.

[0074] The above embodiment specifically describes the process of obtaining a foreground mask image. The following embodiment will describe the process of obtaining a scene mask vector.

[0075] Figure 2 is a schematic diagram of a scene processing module according to an exemplary embodiment of the present disclosure, such as Figure 2 As shown, in the disclosed embodiments, the image generation model can be guided to generate the scene of the image to be generated in a variety of ways. For example, the image generation model can be guided to generate the scene of the image to be generated by text, or by reference images, or both of the above methods can be used to guide the image generation model to generate the scene of the image to be generated.

[0076] If the user selects the text-guided mode, the text information is encoded using a text encoder to obtain a coding vector corresponding to the text information. In this embodiment, the coding vector corresponding to the text information is referred to as a first coding vector. In other words, obtaining a scene coding vector for the scene in which the image to be generated is located may include: in response to receiving text information input by the user, encoding the text information using a text encoder to generate the first coding vector, where the text information is used to describe the scene of the image to be generated.

[0077] The first encoding vector serves as input to the image generation model, guiding the model to generate the corresponding scene. The text information can describe the objects in the scene and the positional relationships between them. For example, the text information could be: "In a room, there is a sofa, a coffee table, a carpet under the coffee table, the curtains are open, and sunlight shines through the window onto the sofa." This means that the text information can be a paragraph described in natural language.

[0078] If the user selects the reference image guidance mode, the reference image is encoded using an image encoder to obtain a coding vector corresponding to the reference image. In this embodiment, the coding vector corresponding to the reference image is referred to as a second coding vector. In other words, obtaining a scene coding vector for the scene in which the image to be generated resides may include: in response to receiving a reference image input by the user, encoding the reference image using an image encoder to generate the second coding vector, where the reference image is used to represent the scene of the image to be generated.

[0079] The second encoded vector serves as the input of the image generation model to guide the model to generate the corresponding scene. Figure 3 is a schematic diagram of a reference image according to an exemplary embodiment of the present disclosure, such as Figure 3 As shown, the reference image contains spatial scene information of the image to be generated, such as objects included in the image to be generated and the positional relationship between the objects.

[0080] It should be noted that the reference image input by the user in this embodiment serves to guide the image generation model to generate a scene that is consistent with the style of the reference image.

[0081] If you choose to guide both methods together, Figure 2As shown, the vectors encoded by the corresponding encoders are then fused by the encoding vector fusion module to obtain a third encoding vector, so that the obtained third encoding vector satisfies both the scene described by the text information and the scene presented in the reference image. That is, obtaining the scene encoding vector for the scene in which the image to be generated resides may include: in response to receiving text information input by a user and a reference image, encoding the text information using a text encoder to generate a first encoding vector, and encoding the reference image using an image encoder to generate a second encoding vector; and fusing the first encoding vector and the second encoding vector to obtain the third encoding vector.

[0082] The present disclosure represents the scene that the user needs to generate in the form of a vector. The vector contains rich information about the scene to be generated, which can provide guidance for the subsequent image generation model to generate the scene, ensuring that the generated scene meets the user's expectations.

[0083] In order to make the scene generated by the image generation model more consistent with user expectations, the present disclosure can further constrain the scene to be generated by obtaining control conditions input by the user. That is, the method can also include: obtaining a control vector corresponding to the control condition input by the user, wherein the control condition includes at least one of a posture graph, a scene layout reference graph, and a light source distribution graph; in this case, fusing the foreground coding vector and the scene coding vector to generate the first target image can include: fusing the foreground coding vector, the scene coding vector, and the control vector to generate the first target image.

[0084] That is, the foreground coding vector, the scene coding vector and the control vector may be input into a pre-trained image generation model, and the image generation model may be used to generate a first target image.

[0085] In some embodiments, acquiring the control vector corresponding to the control condition input by the user includes: acquiring the control condition input by the user, encoding at least one acquired control condition, and generating the control vector.

[0086] Since the control condition may include at least one of a posture graph, a light source distribution graph, and a scene layout reference graph, when multiple control conditions are acquired, the multiple control conditions may be collectively encoded into a control vector.

[0087] User-defined control conditions allow for more precise control over the effects of generated scenes. The introduction of pose maps ensures that models occupy appropriate positions within the scene, making the generated scene more natural and harmonious. Light source distribution maps satisfy users' needs for personalized ambient lighting, allowing for flexible adjustment of light sources and lighting directions according to the needs of different occasions, ultimately presenting a more diverse and richer visual effect. Furthermore, scene layout reference maps enable users to more precisely control the positions of individual objects in the scene based on different layout requirements, thereby generating scene maps that meet specific needs.

[0088] The following embodiments illustrate the process of obtaining each control condition.

[0089] In the case where the foreground mask image contains a target object, the control condition includes a posture diagram. In this case, the control condition for obtaining user input may include: obtaining a first posture diagram drawn by the user according to a first preset rule; or obtaining a second posture diagram selected by the user from a reference posture diagram.

[0090] The foreground in the image to be generated may be a target object such as a model. In the case where the foreground in the image to be generated is a model, the control condition includes a posture diagram. Figure 4 is a schematic diagram of a posture diagram according to an exemplary embodiment of the present disclosure, such as Figure 4 As shown, the posture graph is used to represent the posture of the target object in the foreground mask image through the skeleton. The posture graph helps to recognize and analyze posture, action and posture information by showing the key points and skeleton structure of the human body.

[0091] In one embodiment, the user can select a posture diagram that is the same as the posture of the target object in the foreground from a reference posture diagram. The reference posture diagram is a posture diagram generated based on multiple common postures, such as standing, sitting, and running.

[0092] In another embodiment, when there is no reference posture diagram that is identical to the posture of the target object, the user may draw a posture diagram of the target object in the foreground according to a first preset rule.

[0093] In this embodiment, for the purpose of distinction, the posture diagram drawn by the user is referred to as the first posture diagram, and the posture diagram selected by the user from the reference posture diagram is referred to as the second posture diagram.

[0094] The pose graph helps the model more fully understand the spatial relationship between the model and the scene, such as the model's proportions within the scene, their specific standing position, the floor, and the placement of various objects. This allows the generated scene to blend more naturally with the foreground, avoiding unnatural scenes like the model floating in mid-air or with their head against a wall.

[0095] In the case where the scene coding vector is obtained through text information, the control condition includes a scene layout reference map, which is used to indicate the spatial position of each object in the scene. The scene layout reference map can indicate the spatial position of each object involved in the text information in the overall scene. In this case, the control condition for obtaining user input may include: obtaining a first scene layout reference map drawn by the user according to a second preset rule; or obtaining a second scene layout reference map selected by the user from a scene layout example map.

[0096] Figure 5 This is a schematic diagram of a scene layout reference diagram according to an exemplary embodiment of the present disclosure. Assume that the scene that the user wants to generate is as follows Figure 3 As shown, the user can draw according to the second preset rule as shown in FIG. Figure 5 As shown, the scene layout reference diagram corresponding to the scene that the user expects to generate. The second preset rule is to represent the objects involved in the text information by different colors.

[0097] Taking the text information in the aforementioned embodiment as an example, "In a room, there is a sofa, a coffee table, a carpet under the coffee table, the curtains are open, and sunlight shines through the window onto the sofa", the objects included in the text information include a sofa, a coffee table, a carpet, a window, curtains, etc. In this case, the second preset rule includes using blue to represent a sofa, green to represent a coffee table, dark pink to represent a carpet, red to represent curtains, light gray to represent a window, gray to represent a wall, etc.

[0098] The scene layout reference map can be used to control the placement, orientation, and distance of objects referenced in the text. By adjusting placement, different layouts of the same scene can be constructed to meet the user's personalized needs. Specifically, the scene layout reference map can clarify the spatial position information of each object referenced in the text. When generating scene encoding vectors from text information, combining the scene layout reference map can ensure that the generated image better meets the user's expectations.

[0099] In the case where the control condition includes a posture diagram and / or a scene layout reference diagram, the control condition may further include a light source distribution diagram. In this case, obtaining the control condition input by the user may include: obtaining a light source distribution diagram selected by the user from the reference light source distribution diagram.

[0100] Figure 6 This is a schematic diagram of a light source distribution diagram according to an exemplary embodiment of the present disclosure. In the embodiment of the present disclosure, multiple similar Figure 6 The light source distribution diagram shown is for user selection.

[0101] Users can specify lighting direction and intensity based on their needs, including studio lighting, indoor lighting, and outdoor natural light. By specifying lighting intensity, the contrast between light and dark in the generated scene can be controlled. Specifying lighting direction can change the atmosphere of the generated scene, affecting the shadow information of people and products, thereby conveying the user's desired emotion or expression.

[0102] Through this disclosure, users can specify the distribution of light sources to achieve the desired lighting effect, and can also specify the layout of objects in the scene to ensure that the generated scene meets specific layout requirements.

[0103] Those skilled in the art will appreciate that, in addition to the posture graph, light source distribution graph, and scene layout reference graph, other control conditions can be specified based on actual needs. The combined effect of all these control conditions will guide the final scene generation effect. By setting control conditions, the present disclosure enables the generated scene to be more closely aligned with the user's expectations, providing the user with a richer and more personalized scene generation experience.

[0104] Figure 7 is a framework diagram of an image generating device according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the device includes a foreground processing module, a scene processing module, a control condition processing module, and an image generation module. Because the image generation module generates a scene that meets the user's expectations for the foreground based on a scene coding vector and a control vector, or based on a foreground coding vector, a scene coding vector, and a control vector, the image generation module is also called a scene matching module. A portion of the image generation module's input comes from the foreground processing module, which processes the foreground image to obtain the foreground portion of the image to be generated; a second portion comes from the scene processing module, where an encoder encodes a latent vector containing scene information; and a third portion comes from the control condition processing module, which encodes the various combined control conditions into a control vector. Given the foreground coding vector, scene coding vector, and control vector, the image generation module can generate a scene that meets the user's expectations for the specified foreground. That is, the image generation module generates an image whose proportion of the foreground in the scene conforms to natural laws.

[0105] The present disclosure ensures that the generated scene not only keeps the foreground part expected by the user unchanged, but also effectively generates the complete scene required by the user through a multiple input design.

[0106] After generating the first target image, if the user is not satisfied with the effect presented by the first target image, the user can adjust the first target image by modifying the control conditions. In other words, the method may further include: in response to receiving a modification instruction, displaying the control conditions for generating the first target image; modifying the control conditions based on user operation information; and in response to receiving a generation instruction, generating the corresponding target image based on the modified control conditions.

[0107] By modifying the control conditions, the spatial relationship between objects in the first target image, the lighting effect in the scene, etc. can be adjusted so that the final generated image can present the effect expected by the user.

[0108] When processing a group of images, it is usually required that the scene elements and scene layouts of different images in the same group of images are consistent. Therefore, after generating the first target image, the method may further include: obtaining a foreground coding vector of a second foreground mask image, where the second foreground mask image has the same scene as the first foreground mask image; and generating a second target image based on the foreground coding vector of the second foreground mask image, the scene coding vector, and the control vector.

[0109] After generating the first target image in the same group of images, this embodiment obtains the foreground coding vector of the second foreground mask image in the same group of images, and generates the second target image based on the foreground coding vector of the second foreground mask image, the scene coding vector used when generating the first target image, and the control vector, so as to ensure that the backgrounds between the groups of images remain consistent and avoid inconsistent scenes in a group of photos.

[0110] This embodiment ensures the consistency of the image group scene, that is, by maintaining the same encoding vector and control conditions for the image group scene, a consistent background is generated. In other words, when different operators process the same set of images, the generated scene effect remains consistent, effectively avoiding the uncertainty and various differences that may be introduced by manual operation. By using a fixed scene encoder, the present disclosure can also ensure that the generated image group maintains visual consistency, allowing a group of model images or product images to have a consistent background, facilitating subsequent product display.

[0111] Corresponding to the aforementioned method embodiments, the present disclosure also provides embodiments of a device and a terminal to which the device is applied.

[0112] The following embodiment will introduce an image generation device provided by the present disclosure, the device comprising:

[0113] a foreground processing module, configured to obtain a foreground coding vector of the first foreground mask image;

[0114] A scene processing module, used to obtain a scene coding vector indicating the scene of the image to be generated;

[0115] The image generation module is used to fuse the foreground coding vector and the scene coding vector to generate a first target image.

[0116] In some embodiments, the foreground processing module is specifically configured to obtain a first foreground mask image; and encode the first foreground mask image to obtain a foreground encoding vector of the first foreground mask image.

[0117] In some embodiments, the foreground processing module is specifically used to extract the foreground mask of the foreground object in the first image in response to receiving a first image containing a background input by a user through a pre-trained foreground mask acquisition model, wherein the foreground mask acquisition model is obtained by training a plurality of training sample images, and each training sample image is annotated with the foreground mask of the image and a border surrounding the location of the foreground mask; and a first foreground mask image is generated according to the position of the foreground mask in the first image.

[0118] In some embodiments, the foreground processing module is specifically used to identify the subject where the region of interest is located in the second image in response to receiving a second image containing a background and a specified region of interest input by a user; and segment the subject in the second image to obtain a first foreground mask image.

[0119] In some embodiments, the scene encoding vector includes a first encoding vector; the scene processing module is specifically used to, in response to receiving text information input by a user, encode the text information using a text encoder to generate a first encoding vector, and the text information is used to describe the scene of the image to be generated.

[0120] In some embodiments, the scene coding vector includes a second coding vector; the scene processing module is specifically used to, in response to receiving a reference image input by a user, encode the reference image using an image encoder to generate a second coding vector, and the reference image is used to represent the scene of the image to be generated.

[0121] In some embodiments, the scene coding vector includes a third coding vector; the scene processing module is specifically used to, in response to receiving text information and a reference image input by a user, encode the text information using a text encoder to generate a first coding vector, and encode the reference image using an image encoder to generate a second coding vector; and fuse the first coding vector and the second coding vector to obtain a third coding vector.

[0122] In some embodiments, the device also includes a control condition processing module for obtaining a control vector corresponding to a control condition input by a user, wherein the control condition includes at least one of a posture graph, a scene layout reference graph, and a light source distribution graph; the image generation module is used to fuse the foreground coding vector, the scene coding vector, and the control vector to generate a first target image.

[0123] In some embodiments, the control condition processing module is used to obtain a control condition input by a user; encode at least one obtained control condition, and generate a control vector.

[0124] In some embodiments, when the foreground mask image contains a target object, the control condition includes a posture graph, which is used to represent the posture of the target object in the foreground mask image through a skeleton; the control condition processing module is specifically used to obtain a first posture graph drawn by the user according to a first preset rule; or, obtain a second posture graph selected by the user from a reference posture graph.

[0125] In some embodiments, when the scene coding vector is obtained through text information, the control condition includes a scene layout reference map, which is used to indicate the spatial position of each object in the scene; the control condition processing module is specifically used to obtain a first scene layout reference map drawn by the user according to a second preset rule; or, obtain a second scene layout reference map selected by the user from the scene layout example map.

[0126] In some embodiments, the control condition further includes a light source distribution map; the control condition processing module is specifically configured to obtain a light source distribution map selected by a user from a reference light source distribution map.

[0127] In some embodiments, the device also includes a modification module for, after generating the first target image, displaying the control conditions for generating the first target image in response to receiving a modification instruction; modifying the control conditions according to user operation information; and generating a corresponding target image according to the modified control conditions in response to receiving a generation instruction.

[0128] In some embodiments, the device also includes an image processing module for obtaining a foreground coding vector of a second foreground mask image after generating the first target image, wherein the second foreground mask image has the same scene as the first foreground mask image; and generating a second target image based on the foreground coding vector of the second foreground mask image, the scene coding vector and the control vector.

[0129] The specific implementation of each module in the device has been introduced in the above embodiments and will not be repeated here.

[0130] Figure 8 This is a schematic diagram of the electronic device structure provided by at least one embodiment of the present disclosure. Figure 8 As shown, the electronic device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement an image generation method described in any embodiment of the present disclosure when executing the computer instructions.

[0131] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the image generation methods described in the present disclosure.

[0132] It will be understood by those skilled in the art that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] In the present disclosure, “and / or” means at least one of the two. For example, “A and / or B” includes three solutions: A, B, and “A and B”.

[0134] The above description is merely a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. An image generation method, characterized in that: The method comprises: Obtaining a foreground encoding vector of a first foreground mask image; Obtaining a scene code vector for indicating the scene in which the image to be generated is located; The foreground coding vector and the scene coding vector are fused to generate a first target image.

2. The method according to claim 1, characterized in that The obtaining of the foreground coding vector of the first foreground mask image includes: Obtaining a first foreground mask image; The first foreground mask image is encoded to obtain a foreground encoding vector of the first foreground mask image.

3. The method according to claim 2, characterized in that The obtaining of the first foreground mask image includes: In response to receiving a first image containing a background input by a user, extracting a foreground mask of a foreground object in the first image using a pre-trained foreground mask acquisition model, wherein the foreground mask acquisition model is trained using a plurality of training sample images, each training sample image being annotated with a foreground mask of the image and a bounding box surrounding a location where the foreground mask is located; A first foreground mask image is generated according to the position of the foreground mask in the first image.

4. The method according to claim 2, characterized in that The obtaining of the first foreground mask image includes: In response to receiving a second image including a background and a designated region of interest input by a user, identifying a subject where the region of interest is located in the second image; The subject is segmented in the second image to obtain a first foreground mask image.

5. The method according to claim 1, wherein The scene coding vector includes a first coding vector; The obtaining of a scene coding vector for indicating a scene in which an image to be generated is located includes: In response to receiving text information input by a user, a text encoder is used to encode the text information to generate a first encoding vector, wherein the text information is used to describe a scene of an image to be generated.

6. The method according to claim 1, wherein The scene coding vector includes a second coding vector; The obtaining of a scene coding vector for indicating a scene in which the image to be generated is located includes: In response to receiving a reference image input by a user, an image encoder is used to encode the reference image to generate a second encoding vector, wherein the reference image is used to represent a scene of an image to be generated.

7. The method according to claim 1, characterized in that The scene coding vector includes a third coding vector; The obtaining of a scene coding vector for indicating a scene in which the image to be generated is located includes: In response to receiving text information input by a user and a reference image, encoding the text information using a text encoder to generate a first encoding vector, and encoding the reference image using an image encoder to generate a second encoding vector; The first encoding vector and the second encoding vector are fused to obtain a third encoding vector.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a control vector corresponding to a control condition input by a user, wherein the control condition includes at least one of a posture graph, a scene layout reference graph, and a light source distribution graph; The fusing the foreground coding vector and the scene coding vector to generate a first target image includes: The foreground coding vector, the scene coding vector and the control vector are fused to generate a first target image.

9. The method according to claim 8, characterized in that The obtaining of a control vector corresponding to the control condition input by the user includes: Get control conditions input by the user; The acquired at least one control condition is encoded to generate a control vector.

10. The method according to claim 9, characterized in that In a case where the foreground mask image contains a target object, the control condition includes a posture graph, wherein the posture graph is used to represent a posture of the target object in the foreground mask image through a skeleton; The control conditions for obtaining user input include: Obtaining a first posture diagram drawn by a user according to a first preset rule; or, A second posture image selected by the user from the reference posture image is obtained.

11. The method according to claim 9, characterized in that In the case where the scene coding vector is obtained through text information, the control condition includes a scene layout reference map, where the scene layout reference map is used to indicate the spatial position of each object in the scene; The control conditions for obtaining user input include: Obtaining a first scene layout reference diagram drawn by a user according to a second preset rule; or, Obtain a second scene layout reference image selected by the user from the scene layout example image.

12. The method according to claim 10 or 11, characterized in that The control conditions also include a light source distribution diagram; The control conditions for obtaining user input include: Get the light source distribution map selected by the user from the reference light source distribution map.

13. The method according to claim 8, characterized in that After generating the first target image, the method further includes: In response to receiving the modification instruction, displaying the control conditions for generating the first target image; Modify the control condition according to the user's operation information; In response to receiving the generation instruction, a corresponding target image is generated according to the modified control conditions.

14. The method according to claim 8, characterized in that After generating the first target image, the method further includes: Obtaining a foreground encoding vector of a second foreground mask image, where the second foreground mask image has the same scene as the first foreground mask image; A second target image is generated according to the foreground coding vector of the second foreground mask image, the scene coding vector and the control vector.

15. An image generating device, characterized in that: The device comprises: a foreground processing module, configured to obtain a foreground coding vector of the first foreground mask image; A scene processing module, used to obtain a scene coding vector indicating the scene of the image to be generated; The image generation module is used to fuse the foreground coding vector and the scene coding vector to generate a first target image.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. A terminal device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program and instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images